Image processing device, imaging device, image processing method, and image processing program

ABSTRACT

An image processing unit is configured to generate pixel data for a predetermined color in an intermediate mode implemented based on pixel data obtained from a predetermined region in a light receiving element when an object is captured in a state where infrared light is projected. The image processing unit is configured to generate the pixel data for the predetermined color in a night-vision mode implemented based on pixel data obtained from a wider region than the predetermined region in the light receiving element when the object is captured in the state where the infrared light is projected. The mode stitching unit is configured to switch, depending on conditions, between a state where an image output unit outputs an image signal generated in the intermediate mode and a state where the image output unit outputs an image signal generated in the night-vision mode.

CROSS REFERENCE TO RELATED APPLICATION

This application is based upon and claims the benefit of priority under35 U.S.C. §119 from Japanese Patent Applications No. P2013-242542, filedon Nov. 25, 2013, No. P2013-242544, filed on Nov. 25, 2013, No.2014-011201, filed on Jan. 24, 2014, and No. 2014-078752, filed on Apr.7, 2014, the entire contents of all of which are incorporated herein byreference.

BACKGROUND

The present disclosure relates to an image processing device, an imagingdevice, an image processing method, and an image processing program.

There is known a method for imaging an object under the condition thatalmost no visible light is available, such as during nighttime, byradiating infrared light onto the object from an infrared projector andimaging infrared light reflected by the object. This imaging method iseffective in a case where lighting fixtures for radiating visible lightcannot be used.

However, since an image obtained by imaging the object by this method isa monochromatic image, it is difficult to identify the object from themonochromatic image depending on circumstances. If a color image can becaptured even under the condition that no visible light is available,the performance of identifying the object can be improved. For example,it is expected that surveillance cameras can capture color images underthe condition that no visible light is available in order to improveperformance for identifying objects.

Japanese Unexamined Patent Application Publication No. 2011-050049(Patent Document 1) describes an imaging device capable of capturingcolor images under the condition that no visible light is available. Theimaging device described in Patent Document 1 uses an infraredprojector. Incorporating the technique described in Patent Document 1into a surveillance camera can capture a color image of an object so asto improve the identification of the object.

SUMMARY

For example, visible light is slightly present in outdoor locationsduring the twilight time before sunrise or after sunset or in indoorlocations where illumination is significantly weak. However, since theslightly-present visible light is not sufficient to capture colorimages, objects to be imaged are required to be irradiated with infraredlight for night-vision imaging, as described above.

When infrared light for night-vision imaging is irradiated in a statewhere visible light is present, the visible light and the infrared lightcoexist. Patent Document 1 describes the imaging device on theassumption that imaging is carried out under the condition that novisible light is present. Thus, Patent Document 1 has a problem thatfine color images may not be obtained under the condition that thevisible light and the infrared light for night-vision imaging coexist.

An object of the embodiments is to provide an imaging device capable ofcapturing a fine color image even when visible light and infrared lightfor night-vision imaging coexist, and an image processing device, animage processing method, and an image processing program capable ofgenerating color image signals based on a captured image even whenvisible light and infrared light for night-vision imaging coexist.

A first aspect of the embodiments provides an image processing deviceincluding: an image processing unit including a means of generatingpixel data for a predetermined color in an intermediate mode implementedbased on pixel data obtained from a predetermined region in a lightreceiving element when an object is captured in a state where infraredlight is projected, and a means of generating the pixel data for thepredetermined color in a night-vision mode implemented based on pixeldata obtained from a wider region than the predetermined region in thelight receiving element when the object is captured in the state wherethe infrared light is projected; and a mode stitching unit configured toswitch, depending on a condition, between a state where an image outputunit outputs an image signal generated in the intermediate mode and astate where the image output unit outputs an image signal generated inthe night-vision mode.

A second aspect of the embodiments provides sn imaging device including:the aforementioned image processing device; and an imaging unitconfigured to capture an object.

A third aspect of the embodiments provides an image processing methodincluding the steps of: generating pixel data for a predetermined colorin an intermediate mode implemented based on pixel data obtained from apredetermined region in a light receiving element when an object iscaptured in a state where infrared light is projected; generating thepixel data for the predetermined color in a night-vision modeimplemented based on pixel data obtained from a wider region than thepredetermined region in the light receiving element when the object iscaptured in the state where the infrared light is projected; andswitching, depending on a condition, between a state where an imageoutput unit outputs an image signal generated in the intermediate modeand a state where the image output unit outputs an image signalgenerated in the night-vision mode.

A fourth aspect of the embodiments provides an image processing programrecorded in a non-transitory storage medium implementing meansexecutable by a computer, the means including: an image processing meansincluding a means of generating pixel data for a predetermined color inan intermediate mode implemented based on pixel data obtained from apredetermined region in a light receiving element when an object iscaptured in a state where infrared light is projected, and a means ofgenerating the pixel data for the predetermined color in a night-visionmode implemented based on pixel data obtained from a wider region thanthe predetermined region in the light receiving element when the objectis captured in the state where the infrared light is projected; and amode stitching means of switching, depending on a condition, between astate where an image output unit outputs an image signal generated inthe intermediate mode and a state where the image output unit outputs animage signal generated in the night-vision mode.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing an overall configuration of an imagingdevice of an embodiment.

FIG. 2 is a view showing an example of an array of filter elements in acolor filter used in the imaging device of the embodiment.

FIG. 3 is a characteristic diagram showing spectral sensitivecharacteristics of wavelengths and relative sensitivities of light ofthree primary colors in an imaging unit included in the imaging deviceof the embodiment.

FIG. 4 is a characteristic diagram showing a relationship betweenwavelengths and relative detection rates when multiplying, by a lightreceiving sensitivity of silicon, a reflectance of light of each primarycolor obtained from a particular substance.

FIG. 5 is a block diagram showing a specific configuration example of apre-signal processing unit 52 shown in FIG. 1.

FIG. 6 is a view showing a relationship between exposures and frames ofimage signals when the imaging device of the embodiment is operating ina normal mode.

FIG. 7 is a view for explaining demosaicing when the imaging device ofthe embodiment is operating in the normal mode.

FIG. 8 is a view showing a relationship between exposures and frames ofimage signals when the imaging device of the embodiment is operating inan intermediate mode and in a night-vision mode.

FIG. 9 is a view for explaining pre-signal processing when the imagingdevice of the embodiment is operating in a first intermediate mode.

FIG. 10 is a view for explaining demosaicing when the imaging device ofthe embodiment is operating in the first intermediate mode.

FIG. 11 is a view for explaining pre-signal processing when the imagingdevice of the embodiment is operating in a second intermediate mode.

FIG. 12 is a view for explaining demosaicing when the imaging device ofthe embodiment is operating in the second intermediate mode.

FIG. 13 is a view for explaining processing of adding surrounding pixelswhen the imaging device of the embodiment is operating in thenight-vision mode.

FIG. 14 is a view showing a first weighting example of the processing ofadding the surrounding pixels.

FIG. 15 is a view showing a second weighting example of the processingof adding the surrounding pixels.

FIG. 16 is a view showing a third weighting example of the processing ofadding the surrounding pixels.

FIG. 17 is a view showing frames on which the processing of adding thesurrounding pixels is performed.

FIG. 18 is a view for explaining pre-signal processing when the imagingdevice of the embodiment is operating in a first night-vision mode.

FIG. 19 is a view for explaining demosaicing when the imaging device ofthe embodiment is operating in the first night-vision mode.

FIG. 20 is a view for explaining pre-signal processing when the imagingdevice of the embodiment is operating in a second night-vision mode.

FIG. 21 is a view for explaining demosaicing when the imaging device ofthe embodiment is operating in the second night-vision mode.

FIG. 22 is a view for explaining an example of a mode switch in theimaging device of the embodiment.

FIG. 23 is a view showing conditions of the respective members when theimaging device of the embodiment is set to the respective modes.

FIG. 24 is a partial block diagram showing a first modified example ofthe imaging device of the embodiment.

FIG. 25 is a partial block diagram showing a second modified example ofthe imaging device of the embodiment.

FIG. 26 is a partial block diagram showing a third modified example ofthe imaging device of the embodiment.

FIG. 27 is a flowchart showing an image signal processing method withregard to the mode switch.

FIG. 28 is a flowchart showing a specific processing step in the normalmode shown in step S3 of FIG. 27.

FIG. 29 is a flowchart showing a specific processing step in theintermediate mode shown in step S4 of FIG. 27.

FIG. 30 is a flowchart showing a specific processing step in thenight-vision mode shown in step S5 of FIG. 27.

FIG. 31 is a flowchart showing steps of processing executed by acomputer directed by an image signal processing program with regard tothe mode switch.

FIG. 32 is a characteristic diagram used for considering the spectralsensitive characteristics shown in FIG. 3.

FIG. 33 is a view for explaining exposure amounts when infrared light isradiated in a state where visible light is the predominant light.

FIG. 34 is a view for explaining exposure amounts when infrared light isradiated in a state where infrared light is the predominant light.

FIG. 35 is a schematic explanatory diagram of a determination unit thatimplements a first example of a method of determining a relationshipbetween the amount of visible light and the amount of infrared light.

FIG. 36 is a flowchart showing the first example of the determiningmethod.

FIG. 37 is a flowchart showing processing in which a color gaincontroller controls a color gain setting unit based on the determinationresult.

FIG. 38 is a view for explaining exposure amounts when emitting power ofpart of wavelengths of infrared light is changed in a state whereinfrared light is the predominant light.

FIG. 39 is a schematic explanatory diagram of the determination unitthat implements a second example of the method of determining therelationship between the amount of visible light and the amount ofinfrared light.

FIG. 40 is a view for explaining a difference generated by thedetermination unit when infrared light is radiated in a state wherevisible light is the predominant light.

FIG. 41 is a view for explaining a difference generated by thedetermination unit when infrared light is radiated in a state whereinfrared light is the predominant light.

FIG. 42 is a flowchart showing a second example of the determiningmethod.

FIG. 43 is a flowchart showing an image signal processing methoddepending on the determination of the relationship between therespective light amounts.

FIG. 44 is a flowchart showing steps of processing executed by acomputer directed by an image signal processing program for controllingthe operation of the imaging device including the determination of therelationship between the respective light amounts.

FIG. 45 is a view for explaining sets of color gains that the imagingdevice of the embodiment uses in the respective modes.

DETAILED DESCRIPTION

Hereinafter, an image processing device, an imaging device, an imageprocessing method, and an image processing program according to at leastone embodiment will be explained with reference to appended drawings.

<Configuration of Imaging Device>

First, the entire configuration of the imaging device of the embodimentis explained below with reference to FIG. 1. The imaging device of theembodiment shown in FIG. 1 is capable of capturing images in three modesincluding a normal mode suitable for imaging in a state where sufficientvisible light is present such as during the day, a night-vision modesuitable for imaging in a state where almost no visible light is presentsuch as at night, and an intermediate mode suitable for imaging in astate where visible light is slightly present.

The intermediate mode is a first infrared light radiating mode forimaging while radiating infrared under the condition that the amount ofvisible light is small. The night-vision mode is a second infrared lightradiating mode for imaging while radiating infrared under the conditionthat the amount of visible light is smaller (almost no visible light ispresent).

As shown in FIG. 1, a light indicated by the dashed-dotted linereflected by an object is collected by an optical lens 1. Here, visiblelight enters the optical lens 1 under the condition that visible lightis present sufficiently, and infrared light emitted from an infraredprojector 9 described below and reflected by the object enters theoptical lens 1 under the condition that almost no visible light ispresent.

In the state where visible light is slightly present, mixed lightincluding both the visible light and the infrared light emitted from theinfrared projector 9 and reflected by the object, enters the opticallens 1.

Although FIG. 1 shows only one optical lens 1 for reasons ofsimplification, the imaging device actually includes a plurality ofoptical lenses.

An optical filter 2 is interposed between the optical lens 1 and animaging unit 3. The optical filter 2 includes two members; an infraredcut filter 21 and a dummy glass 22. The optical filter 2 is driven by adrive unit 8 in a manner such that the infrared cut filter 21 isinserted between the optical lens 1 and the imaging unit 3 or such thatthe dummy glass 22 is inserted between the optical lens 1 and theimaging unit 3.

The imaging unit 3 includes an imaging element 31 in which a pluralityof light receiving elements (pixels) are arranged in both the horizontaldirection and the vertical direction, and a color filter 32 in whichfilter elements of red (R), green (G) or blue (B) corresponding to therespective light receiving elements are arranged. The imaging element 31may be either a charge coupled device (CCD) or a complementary metaloxide semiconductor (CMOS).

In the color filter 32, for example, the filter elements of each of R, Gand B are arranged in a pattern called a Bayer array, as shown in FIG.2. The Bayer array is an example of predetermined arrays of the filterelements of R, G and B. In FIG. 2, each of the filter elements of G ineach line held between the filter elements of R is indicated by Gr, andeach of the filter elements of G held between the filter elements of Bis indicated by Gb.

The Bayer array has a configuration in which the horizontal linesalternating the filter elements of R with the filter elements of Gr andthe horizontal lines alternating the filter elements of B with thefilter elements of Gb are aligned alternately with each other in thevertical direction.

FIG. 3 shows spectral sensitive characteristics of wavelengths andrelative sensitivities of R light, G light and B light in the imagingunit 3. The maximum value of the relative sensitivities is normalizedto 1. When the imaging device is operated in the normal mode, infraredlight having a wavelength of 700 nm or greater is required to be blockedin order to capture fine color images with visible light.

The drive unit 8 is thus controlled by a controller 7 to drive theoptical filter 2 in such a manner as to insert the infrared cut filter21 between the optical lens 1 and the imaging unit 3.

As is apparent from FIG. 3, the imaging unit 3 shows the sensitivitiesin the area where the infrared light having the wavelength of 700 nm orgreater is present. Therefore, when the imaging device is operated inthe intermediate mode or in the night-vision mode, the drive unit 8 iscontrolled by the controller 7 to drive the optical filter 2 in such amanner as to remove the infrared cut filter 21 from between the opticallens 1 and the imaging unit 3 and insert the dummy glass 22therebetween. When the dummy glass 22 is inserted between the opticallens 1 and the imaging unit 3, the infrared light having the wavelengthof 700 nm or greater is not blocked. Thus, the imaging device can obtaininformation of each of R, G and B by using the sensitivities in the ovalregion surrounded by the broken line in FIG. 3. The reason the dummyglass 22 is inserted is to conform the optical path length obtained whenthe dummy glass 22 is used to the optical path length obtained when theinfrared cut filter 21 is used.

The infrared projector 9 includes projecting portions 91, 92 and 93 forprojecting infrared light with wavelengths IR1, IR2 and IR3,respectively. In the case of the intermediate mode or the night-visionmode, a projection controller 71 in the controller 7 controls theprojecting portions 91, 92 and 93 so as to selectively project theinfrared light with the respective wavelengths IR1, IR2 and IR3 in atime division manner.

Here, a silicon wafer is used in the imaging element 31. FIG. 4 shows arelationship between wavelengths and relative detection rates when areflectance at each wavelength is multiplied by a light receivingsensitivity of silicon in a case where a material consisting of each ofthe colors R, G and B is irradiated with white light. The maximum valueof the relative detection rates in FIG. 4 is also normalized to 1.

For example, as shown in FIG. 4, in the infrared light area, thereflected light with the wavelength of 780 nm has a strong correlationwith the reflected light of the material with color R, the reflectedlight with the wavelength of 870 nm has a strong correlation with thereflected light of the material with color B, and the reflected lightwith the wavelength of 940 nm has a strong correlation with thereflected light of the material with color G.

Thus, according to the present embodiment, the wavelengths IR1, IR2 andIR3 of infrared light projected from the projecting portions 91, 92 and93 are set to 780 nm, 940 nm and 870 nm, respectively. These values areexamples for the wavelengths IR1, IR2 and IR3, and other wavelengthsother than 780 nm, 940 nm and 870 nm may also be employed.

The projecting portion 91 radiates the infrared light with thewavelength IR1 on an object, and an image signal obtained, in a mannersuch that light reflected by the object is captured, is assigned to an Rsignal. The projecting portion 93 radiates the infrared light with thewavelength IR2 on the object, and an image signal obtained, in a mannersuch that light reflected by the object is captured, is assigned to a Gsignal. The projecting portion 92 radiates the infrared light with thewavelength IR3 on the object, and an image signal obtained, in a mannersuch that light reflected by the object is captured, is assigned to a Bsignal.

Accordingly, even in the intermediate mode or in the night-vision mode,a color similar to that obtained when the object is imaged in the normalmode in the state where visible light is present, can be reproducedtheoretically.

Alternatively, the wavelength IR1 of 780 nm may be assigned to the Rlight, the wavelength IR3 of 870 nm may be assigned to the G light, andthe wavelength IR2 of 940 nm may be assigned to the B light, although inthis case the color image would possess a color tone different from theactual color tone of the object. The wavelengths IR1, IR2 and IR3 may beassigned optionally to the R light, the G light and the B light.

According to the present embodiment, the wavelengths IR1, IR2 and IR3are assigned to the R light, the G light and the B light, respectively,by which the color tone of the object can be reproduced most finely.

The controller 7 controls the imaging unit 3, the respective componentsin an image processing unit 5, and an image output unit 6. Image signalscaptured by the imaging unit 3 are subjected to A/D conversion by an A/Dconverter 4 and then input into the image processing unit 5. Theinternal configuration and operation of the image processing unit 5 willbe explained below.

The image output unit 6 includes a color gain setting unit 62 thatmultiplies data for three primary colors R, G and B described below byeach predetermined color gain. The specific operation of the color gainsetting unit 62 will be explained below. The color gain setting unit 62may be provided in the image processing unit 5.

The imaging unit 3 and the A/D converter 4 may be integrated together.The image processing unit 5 and the controller 7 may be integratedtogether.

The controller 7 includes a mode switching unit 72 that switches amongthe normal mode, the intermediate mode and the night-vision mode. Themode switching unit 72 switches the operations in the image processingunit 5 as appropriate, as described below, corresponding to the normalmode, the intermediate mode and the night-vision mode.

The controller 7 further includes a determination unit 78 that analyzesthe relationship between the amount of environmental visible light andthe amount of infrared light, and a color gain controller 79 thatcontrols the color gain setting unit 62 to vary each color gain by whichthe data for the respective three primary colors is multiplied. Here,the infrared light is mostly composed of light emitted from the infraredprojector 9 and reflected by an object to be captured.

The determination unit 78 is only required to, for example, determinewhether the amount of visible light is greater than the amount ofinfrared light so that the visible light is the predominant light andthe infrared light is the subordinate light, or whether the amount ofinfrared light is greater than the amount of visible light so that theinfrared light is the predominant light and the visible light is thesubordinate light. The determination unit 78 may analyze therelationship between the amount of environmental visible light and theamount of infrared light in such a manner as to calculate a ratio of thespecific light amounts.

Here, the ratio of light amounts is not necessarily the ratio of theamount of visible light and the amount of infrared light itself as longas it represents a numerical value that varies depending on therelationship (the ratio or the like) between the amount of environmentalvisible light and the amount of infrared light.

When calculating the numerical value so as to analyze the relationshipbetween the amount of environmental visible light and the amount ofinfrared light, it is not required to determine which of visible lightand infrared light is present predominantly or subordinately.Hereinafter, for convenience of explanation, the relationship betweenthe amount of environmental visible light and the amount of infraredlight may simply be referred to as a “superior-subordinaterelationship”.

The color gain controller 79 controls the color gain setting unit 62 tovary each color gain by which the data for the respective three primarycolors is multiplied according to the relationship between the amount ofenvironmental visible light and the amount of infrared light analyzed bythe determination unit 78.

The image processing unit 5 includes switches 51 and 53, a pre-signalprocessing unit 52, and a demosaicing unit 54. The switches 51 and 53may be physical switches or may be logical switches for switching thepre-signal processing unit 52 between an active state and an inactivestate. The controller 7 receives an imaging signal input from the imageprocessing unit 5 in order to detect brightness of an image beingcaptured.

Image data input into the pre-signal processing unit 52 is also inputinto the determination unit 78. The determination unit 78 analyzes thesuperior-subordinate relationship between the light amounts based on theimage data input into the pre-signal processing unit 52.

The pre-signal processing unit 52 may possess a function of thedetermination unit 78. The pre-signal processing unit 52 possessing thefunction of the determination unit 78 notifies the color gain controller79 of the determination result of the superior-subordinate relationshipbetween the light amounts.

As shown in FIG. 5, the pre-signal processing unit 52 includes asurrounding pixel adding unit 521, a same-position pixel adding unit522, and a synthesizing unit 523.

The image processing unit 5 generates data for the respective threeprimary colors R, G and B and supplies the data to the image output unit6. The image output unit 6 outputs the data for the three primary colorsin a predetermined format to a display unit or the like (not shown inthe drawing).

The image output unit 6 may directly output signals of the three primarycolors R, G and B, or may convert the signals of the three primarycolors R, G and B into luminance signals and color signals (or colordifference signals) before outputting. The image output unit 6 mayoutput composite image signals. The image output unit 6 may outputdigital image signals or output image signals converted into analogsignals by a D/A converter.

When the image signals in a predetermined format are output from theimage processing unit 5, the color gain setting unit 62 multiplies thedata for the respective three primary colors R, G and B by therespective predetermined color gains in order to adjust white balance.Note that the color gain setting unit 62 may multiply the data by eachcolor gain in order to reproduce an image at a predetermined colortemperature, instead of multiplying in order to adjust the whitebalance.

The color gain setting unit 62 holds at least two sets of color gainsused for multiplying the data for the three primary colors R, G and B.The color gains in one set used for the data for the respective colorsR, G and B are all different from the color gains in the other set.

When the imaging device is set to the intermediate mode by the modeswitching unit 72, the color gain controller 79 controls the color gainsetting unit 62 to select one of the sets of color gains used formultiplying the three primary color data. The color gain setting unit 62multiplies the three primary color data by the color gains of theselected set.

When the imaging device is set to the normal mode or the night-visionmode by the mode switching unit 72, the color gain setting unit 62multiplies the three primary color data by a set of color gains fixeddifferently in each mode.

Next, the operations of each of the normal mode, the intermediate modeand the night-vision mode are explained in more detail.

<Normal Mode>

In the normal mode, the controller 7 directs the drive unit 8 to insertthe infrared cut filter 21 between the optical lens 1 and the imagingunit 3. The projection controller 71 turns off the infrared projector 9to stop projecting infrared light.

Image signals captured by the imaging unit 3 are converted into imagedata as digital signals by the A/D converter 4 and then input into theimage processing unit 5. In the normal mode, the mode switching unit 72connects the switches 51 and 53 to the respective terminals Tb.

Item (a) of FIG. 6 shows exposures Ex1, Ex2, Ex3, . . . of the imagingunit 3. Although the actual exposure time varies depending on conditionssuch as a shutter speed, each of the exposures Ex1, Ex2, Ex3 shows themaximum exposure time.

Item (b) of FIG. 6 shows the timing at which each of frames of the imagesignals is obtained. Frame F0 of the image signals is obtained based onan exposure (not shown in the drawing) prior to the exposure Ex1 after apredetermined period of time. Frame F1 of the image signals is obtainedbased on the exposure Ex1 after a predetermined period of time. Frame F2of the image signal is obtained based on the exposure Ex2 after apredetermined period of time. The same operations are repeated after theexposure Ex3. A frame frequency of the image signals is, for example, 30frames per second.

The frame frequency of the image signals may be set to, for example, 30frames per second or 60 frames per second in the NTSC format, or 25frames per second or 50 frames per second in the PAL format.Alternatively, the frame frequency of the image signals may be 24 framesper second which is used for movies.

The image data of each frame output from the A/D converter 4 is input tothe demosaicing unit 54 via the switches 51 and 53. The demosaicing unit54 subjects the image data of each input frame to demosaicing. The imageprocessing unit 5 subjects the data to other types of image processingin addition to the demosaicing and then outputs the data for the threeprimary colors R, G and B.

The demosaicing in the demosaicing unit 54 is explained below withreference to FIG. 7. Item (a) of FIG. 7 shows an arbitrary frame Fm ofimage data. The frame Fm is composed of pixels in an effective imageperiod. The number of the pixels is, for example, 640 horizontal pixelsand 480 vertical pixels in the VGA standard. For reasons ofsimplification, the number of the pixels in the frame Fm is greatlydecreased so as to schematically show the frame Fm.

The image data generated by the imaging unit 3 having the Bayer array isdata in which pixel data for R, G and B are mixed in the frame Fm. Thedemosaicing unit 54 computes pixel data for R for pixel positions whereno pixel data for R is present by use of the surrounding pixel data forR so as to generate interpolated pixel data Ri for R. The demosaicingunit 54 generates R frame FmR in which all pixels in one frame shown initem (b) of FIG. 7 are composed of the pixel data for R.

The demosaicing unit 54 computes pixel data for G for pixel positionswhere no pixel data for G is present by use of the surrounding pixeldata for G so as to generate interpolated pixel data Gi for G. Thedemosaicing unit 54 generates G frame FmG in which all pixels in oneframe shown in item (c) of FIG. 7 are composed of the pixel data for G.

The demosaicing unit 54 computes pixel data for B for pixel positionswhere no pixel data for B is present by use of the surrounding pixeldata for B so as to generate interpolated pixel data Bi for B. Thedemosaicing unit 54 generates B frame FmB in which all pixels in oneframe shown in item (d) of FIG. 7 are composed of the pixel data for B.

The demosaicing unit 54 is only required to use at least the pixel datafor R when interpolating the pixel data for R, use at least the pixeldata for G when interpolating the pixel data for G, and use at least thepixel data for B when interpolating the pixel data for B. Alternatively,the demosaicing unit 54 may interpolate the pixel data for each of R, Gand B to be generated by use of the pixel data of the different colorsin order to improve the accuracy of the interpolation.

Since the imaging unit 3 further includes pixels outside the effectiveimage period, pixel data for each of R, G and B can be interpolated withregard to the pixels located along the edges of top and bottom, left andright.

The R frame FmR, the G frame FmG and the B frame FmB generated by thedemosaicing unit 54 are output as the data for the three primary colorsR, G and B. Although the pixel data for each of R, G and B was explainedper frame in FIG. 7 for ease of explanation, the pixel data for each ofR, G and B is actually output sequentially per pixel.

<Intermediate Mode: First Intermediate Mode>

In the intermediate mode (first intermediate mode and secondintermediate mode described below), the controller 7 directs the driveunit 8 to insert the dummy glass 22 between the optical lens 1 and theimaging unit 3. The projection controller 71 turns on the infraredprojector 9 to project infrared light. The mode switching unit 72connects the switches 51 and 53 to the respective terminals Ta.

Item (a) of FIG. 8 shows a state where infrared light is projected fromthe infrared projector 9. The controller 7 divides one frame period ofthe normal mode into three so as to control, for example, the projectingportions 91, 92 and 93 to sequentially project infrared light in thisorder.

In the example of item (a) of FIG. 8, the infrared light with thewavelength IR1 (780 nm) is radiated on the object in the first ⅓ periodof the one frame. The infrared light with the wavelength IR2 (940 nm) isradiated on the object in the second ⅓ period of the one frame. Theinfrared light with the wavelength IR3 (870 nm) is radiated on theobject in the last ⅓ period of the one frame. The order of radiation ofthe infrared light with the respective wavelengths IR1, IR2 and IR3 isoptional. However, for the following reasons, the case in which thewavelength IR2 is located in the middle, and the wavelengths IR1, IR2and IR3 are arranged in this order, is most preferable.

In the intermediate mode and the night-vision mode, as described below,image signals of one frame are generated based on imaging signalsgenerated three times by the imaging unit 3 in the state where theinfrared light of each of the wavelengths IR1, IR2 and IR3 is projected.Therefore, when a moving object is imaged, a color shift or a blur onthe outline may be caused.

In general, G signals among signals R, G and B have the greatestinfluence on luminance signals. Thus, the infrared light with thewavelength IR2 for generating the image signals (imaging signals)assigned to the G signals are preferably arranged in the middle so thatthe infrared light with the wavelengths IR1 and IR3 is respectivelyarranged in front of and behind the infrared light with the wavelengthIR2.

As shown in item (b) of FIG. 8, exposure Ex1R which has a strongcorrelation with R light is executed by the imaging unit 3 at the pointwhere the infrared light with the wavelength IR1 is being projected.Exposure Ex1G which has a strong correlation with G light is executed bythe imaging unit 3 at the point where the infrared light with thewavelength IR2 is being projected. Exposure Ex1B which has a strongcorrelation with B light is executed by the imaging unit 3 at the pointwhere the infrared light with the wavelength IR3 is being projected.

Note that, since an image is captured in the intermediate mode in astate where visible light is slightly present, visible light andinfrared light projected from the infrared projector 9 coexist.Therefore, in the intermediate mode, exposures Ex1R, Ex1G, Ex1B, Ex2R,Ex2G, Ex2B, etc., are each obtained in a manner such that exposure ofvisible light and exposure of infrared light are combined together. Asshown in item (c) of FIG. 8, frame F1IR1 corresponding to the exposureEx1R, frame F1IR2 corresponding to the exposure Ex1G and frame F1IR3corresponding to the exposure Ex1B are obtained based on the exposuresEx1R, Ex1G and Ex1B after a predetermined period of time.

Further, frame F2IR1 corresponding to the exposure Ex2R, frame F2IR2corresponding to the exposure Ex2G and frame F2IR3 corresponding to theexposure Ex2B are obtained based on the exposures Ex2R, Ex2G and Ex2Bafter a predetermined period of time. The same operations are repeatedafter the exposures Ex3R, Ex3G, and Ex3B.

The frame frequency of the imaging signals in item (c) of FIG. 8 is 90frames per second. In the intermediate mode, one frame of the imagingsignals in the normal mode is subjected to time division so as toproject the infrared light with the respective wavelengths IR1 to IR3.Thus, in order to output the imaging signals in the same format as thenormal mode, the frame frequency of the imaging signals in item (c) ofFIG. 8 is three times as many as that in the normal mode.

As described below, based on the imaging signals of the three frames initem (c) of FIG. 8, one frame of imaging signals having a framefrequency of 30 frames per second as shown in item (d) of FIG. 8 isgenerated. For example, frame F1IR is generated based on the framesF1IR1, F1IR2 and F1IR3, and frame F2IR is generated based on the framesF2IR1, F2IR2 and F2IR3.

The operation of generating the imaging signals of each frame in item(d) of FIG. 8 in the intermediate mode based on the imaging signals ofthe three frames in item (c) of FIG. 8, is explained in detail below.

The image data for the respective frames corresponding to the imagingsignals shown in item (c) of FIG. 8 output from the A/D converter 4 isinput into the pre-signal processing unit 52 via the switch 51.

Pre-signal processing in the pre-signal processing unit 52 is explainedbelow with reference to FIG. 9. Item (a) of FIG. 9 shows an arbitraryframe FmIR1 of image data generated at the point where the infraredlight with the wavelength IR1 is being projected. The pixel data foreach of R, B, Gr and Gb in the frame FmIR1 is added with an index “1”indicating that all data is generated in the state where the infraredlight with the wavelength IR1 is projected.

Item (b) of FIG. 9 shows an arbitrary frame FmIR2 of image datagenerated at the point where the infrared light with the wavelength IR2is being projected. The pixel data for each of R, B, Gr and Gb in theframe FmIR2 is added with an index “2” indicating that all data isgenerated in the state where the infrared light with the wavelength IR2is projected.

Item (c) of FIG. 9 shows an arbitrary frame FmIR3 of image datagenerated at the point where the infrared light with the wavelength IR3is being projected. The pixel data for each of R, B, Gr and Gb in theframe FmIR3 is added with an index “3” indicating that all data isgenerated in the state where the infrared light with the wavelength IR3is projected.

Since the frame FmIR1 shown in item (a) of FIG. 9 includes the imagedata generated in the state where the infrared light with the wavelengthIR1 having a strong correlation with R light is projected, the pixeldata for R is pixel data corresponding to the projected infrared light,and the pixel data for B and G are pixel data not corresponding to theprojected infrared light. The hatching added to the pixel data for eachof B, Gr and Gb represents that the pixel data does not correspond tothe projected infrared light.

Since the frame FmIR2 shown in item (b) of FIG. 9 includes the imagedata generated in the state where the infrared light with the wavelengthIR2 having a strong correlation with G light is projected, the pixeldata for G is pixel data corresponding to the projected infrared light,and the pixel data for R and B are pixel data not corresponding to theprojected infrared light. The hatching added to the pixel data for eachof R and B represents that the pixel data does not correspond to theprojected infrared light.

Since the frame FmIR3 shown in item (c) of FIG. 9 includes the imagedata generated in the state where the infrared light with the wavelengthIR3 having a strong correlation with B light is projected, the pixeldata for B is pixel data corresponding to the projected infrared light,and the pixel data for R and G are pixel data not corresponding to theprojected infrared light. The hatching added to the pixel data for eachof R, Gr and Gb represents that the pixel data does not correspond tothe projected infrared light.

The same-position pixel adding unit 522 in the pre-signal processingunit 52 individually adds the pixel data for each of R, Gr, Gb and Blocated at the same pixel positions according to the following formulae(1) to (3) so as to generate added pixel data R123, Gr123, Gb123, andB123. In the intermediate mode, the surrounding pixel adding unit 521 inthe pre-signal processing unit 52 is inactive.

R123=ka×R1+Kb×R2+kc×R3   (1)

G123=kd×G1+Ke×G2+kf×G3   (2)

B123=kg×B1+Kh×B2+ki×R3   (3)

In the formulae (1) to (3), R1, G1 and B1 are pixel data for R, G and Bin the frame FmIR1, R2, G2 and B2 are pixel data for R, G and B in theframe FmIR2, and R3, G3 and B3 are pixel data for R, G and B in theframe FmIR3. In addition, ka to ki are predetermined coefficients. Thedata G123 in the formula (2) is either Gr123 or Gb123.

The same-position pixel adding unit 522 adds the hatched pixel data foreach of R, Gr, Gb and B to the pixel data for each of R, Gr, Gb and Blocated at the same pixel positions not hatched.

In particular, the same-position pixel adding unit 522 adds, to thepixel data for R located in the frame FmIR1, the pixel data for Rlocated at the same pixel positions in each of the frames FmIR2 andFmIR3 so as to generate the added pixel data R123 according to theformula (1). That is, the same-position pixel adding unit 522 only usesthe pixel data in the region corresponding to the red color filter inthe light receiving elements and generates the added pixel data R123 forred.

The same-position pixel adding unit 522 adds, to the pixel data for Gr,Gb located in the frame FmIR2, the pixel data for Gr, Gb located at thesame pixel positions in each of the frames FmIR1 and FmIR3 so as togenerate the added pixel data G123 according to the formula (2). Thatis, the same-position pixel adding unit 522 only uses the pixel data inthe region corresponding to the green color filter in the lightreceiving elements and generates the added pixel data G123 for green.

The same-position pixel adding unit 522 adds, to the pixel data for Blocated in the frame FmIR3, the pixel data for B located at the samepixel positions in each of the frames FmIR1 and FmIR2 so as to generatethe added pixel data B123 according to the formula (3). That is, thesame-position pixel adding unit 522 only uses the pixel data in theregion corresponding to the blue color filter in the light receivingelements and generates the added pixel data B123 for blue.

The synthesizing unit 523 in the pre-signal processing unit 52 generatesframe FmIR123 of synthesized image signals shown in item (d) of FIG. 9based on the respective added pixel data R123, Gr123, Gb123 and B123generated at the respective pixel positions.

More particularly, the synthesizing unit 523 selects the added pixeldata R123 in the frame FmIR1, the added pixel data Gr123 and Gb123 inthe frame FmIR2, and the added pixel data B123 in FmIR3 and synthesizesthe respective added pixel data. The synthesizing unit 523 thusgenerates the frame FmIR123 of the synthesized image signals.

As described above, the synthesizing unit 523 generates the frameFmIR123 in which the respective added pixel data R123, Gr123, Gb123 andB123 are arranged so as to have the same array as the filter elements inthe color filter 32.

In the first intermediate mode, the image data in the frame FmIR123 aregenerated in such a manner as to use the pixel data not hatched and thepixel data hatched.

The reason the same-position pixel adding unit 522 adds the respectivepixel data located at the same pixel positions is that, since an imageis captured in the intermediate mode in the state where visible light ispresent, although the amount thereof is small, the hatched pixel datacontains the components of the respective colors based on the exposureby the visible light. Therefore, the respective pixel data located atthe same pixel positions are added to each other so that the sensitivityto the respective colors can be improved.

When the amount of visible light is relatively large in the state wherevisible light and infrared light coexist, the exposure by the visiblelight is predominant. In such a case, the image data in the frameFmIR123 mainly contains the components based on the image signalsexposed by the visible light. When the amount of infrared light isrelatively large in the state where infrared light and visible lightcoexist, the exposure by the infrared light is predominant. In such acase, the image data in the frame FmIR123 mainly contains the componentsbased on the image signals exposed by the infrared light.

When the amount of visible light is relatively small, the coefficientska, kb and kc in the formula (1) preferably fulfill the relationship ofka>kb, kc, the coefficients kd, ke and kf in the formula (2) preferablyfulfill the relationship of kf>kd, ke, and the coefficients kg, kh andki in the formula (3) preferably fulfill the relationship of kh>kg, ki.This is because the wavelength IR1 has a strong correlation with the Rlight, the wavelength IR2 has a strong correlation with the G light, andthe wavelength IR3 has a strong correlation with the B light.

Accordingly, the pixel data for R can be the main data in the frameFmIR1, the pixel data for G can be the main data in the frame FmIR2, andthe pixel data for B can be the main data in the frame FmIR3.

The image data in the frame FmIR123 output from the pre-signalprocessing unit 52 is input into the demosaicing unit 54 via the switch53. The demosaicing unit 54 subjects the input image data in the frameFmIR123 to demosaicing in the same manner as the normal mode. The imageprocessing unit 5 subjects the image data to other types of imageprocessing in addition to the demosaicing and then outputs the data forthe three primary colors R, G and B.

The demosaicing in the demosaicing unit 54 is explained below withreference to FIG. 10. Item (a) of FIG. 10 shows the frame FmIR123. Thedemosaicing unit 54 computes pixel data for R for pixel positions whereno pixel data for R is present by use of the surrounding pixel data forR so as to generate interpolated pixel data R123 i for R. Thedemosaicing unit 54 generates R frame FmIR123R in which all pixels inone frame shown in item (b) of FIG. 10 are composed of the pixel datafor R.

The demosaicing unit 54 computes pixel data for G for pixel positionswhere no pixel data for G is present by use of the surrounding pixeldata for G so as to generate interpolated pixel data G123 i for G. Thedemosaicing unit 54 generates G frame FmIR123G in which all pixels inone frame shown in item (c) of FIG. 10 are composed of the pixel datafor G.

The demosaicing unit 54 computes pixel data for B for pixel positionswhere no pixel data for B is present by use of the surrounding pixeldata for B so as to generate interpolated pixel data B123 i for B. Thedemosaicing unit 54 generates B frame FmIR123B in which all pixels inone frame shown in item (d) of FIG. 10 are composed of the pixel datafor B.

As is apparent from the operation of the demosaicing unit 54 in thenormal mode shown in FIG. 7 and the operation of the demosaicing unit 54in the intermediate mode shown in FIG. 10, both operations aresubstantially the same. Thus, the operation of the demosaicing unit 54does not differ between the normal mode and the intermediate mode.

The pre-signal processing unit 52 is only required to be activated inthe intermediate mode except for the surrounding pixel adding unit 521,while the pre-signal processing unit 52 is inactivated in the normalmode. The normal mode and the intermediate mode may share the signalprocessing unit such as the demosaicing unit 54 in the image processingunit 5.

<Intermediate Mode: Second Intermediate Mode>

Operations in the second intermediate mode are explained below withreference to FIG. 11 and FIG. 12. Note that the same operations as thosein the first intermediate mode are not repeated in the secondintermediate mode. Here, the frame FmIR1, the frame FmIR2 and the frameFmIR3 shown in items (a) to (c) in FIG. 11 are the same as the frameFmIR1, the frame FmIR2 and the frame FmIR3 shown in items (a) to (c) inFIG. 9.

The synthesizing unit 523 selects pixel data R1 for R in the frameFmIR1, pixel data Gr2 and Gb2 for G in the frame FmIR2, and pixel dataB3 for B in FmIR3 and synthesizes the respective pixel data. Thesynthesizing unit 523 thus generates frame FmIR123′ of the synthesizedimage signals shown in item (d) of FIG. 11.

That is, the frame FmIR123′ is image data in which the pixel data for R,Gr, Gb and B not hatched in each of the frames FmIR1, FmIR2 and FmIR3are collected in one frame.

Thus, the frame FmIR123′ contains the pixel data for red only using thepixel data in the region corresponding to the red color filter in thestate where the infrared light with the wavelength IR1 is projected, thepixel data for green only using the pixel data in the regioncorresponding to the green color filter in the state where the infraredlight with the wavelength IR2 is projected, and the pixel data for blueonly using the pixel data in the region corresponding to the blue colorfilter in the state where the infrared light with the wavelength IR3 isprojected.

As described above, the synthesizing unit 523 generates the frameFmIR123′ in which the respective pixel data R1, Gr2, Gb2 and B3 arearranged so as to have the same array as the filter elements in thecolor filter 32.

In the second intermediate mode, the same-position pixel adding unit 522defines the coefficient Ka in the formula (1) as 1 and the othercoefficients Kb and Kc as 0, defines the coefficient ke in the formula(2) as 1 and the other coefficients kd and kf as 0, and defines thecoefficient ki in the formula (3) as 1 and the other coefficients kg andkh as 0.

Therefore, the value of the pixel data for R in the frame FmIR1, thevalues of the pixel data for Gr and Gb in the frame FmIR2 and the valueof the pixel data for B in the frame FmIR3 each remain as is.

Accordingly, the synthesizing unit 523 can generate the frame FmIR123′by selecting the pixel data for R in the frame FmIR1, the pixel data forGr and Gb in the frame FmIR2 and the pixel data for B in the frameFmIR3, in the same manner as the operations in the first intermediatemode.

In the second intermediate mode, the pre-signal processing unit 52 onlyuses the pixel data (the pixel data not hatched) generated in the statewhere the infrared light for generating the pixel data with the samecolor is projected so as to generate the frame FmIR123′.

According to the second intermediate mode, although the sensitivity orcolor reproduction performance decreases compared with the firstintermediate mode, the calculation processing can be simplified or theframe memory can be reduced.

The demosaicing in the demosaicing unit 54 is explained below withreference to FIG. 12. Item (a) of FIG. 12 shows the frame FmIR123′. Thedemosaicing unit 54 computes pixel data for R for pixel positions whereno pixel data for R is present by use of the surrounding pixel data forR so as to generate interpolated pixel data R1 i for R. The demosaicingunit 54 generates R frame FmIR123′ R in which all pixels in one frameshown in item (b) of FIG. 12 are composed of the pixel data for R.

The demosaicing unit 54 computes pixel data for G for pixel positionswhere no pixel data for G is present by use of the surrounding pixeldata for G so as to generate interpolated pixel data G2 i for G. Thedemosaicing unit 54 generates G frame FmIR123′ G in which all pixels inone frame shown in item (c) of FIG. 12 are composed of the pixel datafor G.

The demosaicing unit 54 computes pixel data for B for pixel positionswhere no pixel data for B is present by use of the surrounding pixeldata for B so as to generate interpolated pixel data B3 i for B. Thedemosaicing unit 54 generates B frame FmIR123′ B in which all pixels inone frame shown in item (d) of FIG. 12 are composed of the pixel datafor B.

Accordingly, in the intermediate mode, the pixel data for red isgenerated from the pixel data obtained from the region corresponding tothe red color filter in the light receiving elements, the pixel data forgreen is generated from the pixel data obtained from the regioncorresponding to the green color filter in the light receiving elements,and the pixel data for blue is generated from the pixel data obtainedfrom the region corresponding to the blue color filter in the lightreceiving elements.

<Night-Vision Mode: First Night-Vision Mode>

In the night-vision mode (first night-vision mode and secondnight-vision mode described below), the controller 7 directs the driveunit 8 to insert the dummy glass 22 between the optical lens 1 and theimaging unit 3, as in the case of the intermediate mode. The projectioncontroller 71 turns on the infrared projector 9 to project infraredlight. The mode switching unit 72 connects the switches 51 and 53 to therespective terminals Ta.

The general operations in the night-vision mode are the same as thoseshown in FIG. 8. However, since an image is captured in the night-visionmode in a state where almost no visible light is present, the exposuresEx1R, Ex1G, Ex1B, Ex2R, Ex2G, Ex2B, etc., shown in item (b) of FIG. 8are assumed to be exposure only by infrared light.

Under the condition that there is almost no visible light but onlyinfrared light, the characteristics of the respective filter elements inthe color filter 32 differ from each other. Thus, the imaging unit 3 canbe considered as a single-color imaging device.

Therefore, in the night-vision mode, the surrounding pixel adding unit521 in the pre-signal processing unit 52 adds surrounding pixel data toall pixel data in order to improve the sensitivity of infrared light.

More particularly, when the R pixel is the target pixel as shown in item(a) of FIG. 13, the surrounding pixel adding unit 521 adds, to the pixeldata for R as the target pixel, the pixel data of the surrounding eightpixels of G (Gr, Gb) and B.

That is, while the pixel data for red is generated from the pixel dataobtained from the region corresponding to the red color filter in thelight receiving elements in the intermediate mode, the pixel data forred is generated, in the night-vision mode, from the pixel data obtainedfrom a wider region than the region in the intermediate mode. Therespective examples shown in items (a) to (d) of FIG. 13 use the pixeldata obtained from the region of the nine pixels including the targetpixel.

When the Gr pixel is the target pixel as shown in item (b) of FIG. 13,the surrounding pixel adding unit 521 adds, to the pixel data for Gr asthe target pixel, the pixel data of the surrounding eight pixels of R,Gb and B. When the Gb pixel is the target pixel as shown in item (c) ofFIG. 13, the surrounding pixel adding unit 521 adds, to the pixel datafor Gb as the target pixel, the pixel data of the surrounding eightpixels of R, Gr and B.

That is, while the pixel data for green is generated from the pixel dataobtained from the region corresponding to the green color filter in thelight receiving elements in the intermediate mode, the pixel data forgreen is generated, in the night-vision mode, from the pixel dataobtained from a wider region than the region in the intermediate mode.

When the B pixel is a target pixel as shown in item (d) of FIG. 13, thesurrounding pixel adding unit 521 adds, to the pixel data for B as thetarget pixel, the pixel data of the surrounding eight pixels of R and G.

That is, while the pixel data for blue is generated from the pixel dataobtained from the region corresponding to the blue color filter in thelight receiving elements in the intermediate mode, the pixel data forblue is generated, in the night-vision mode, from the pixel dataobtained from a wider region than the region in the intermediate mode.

The surrounding pixel adding unit 521 may simply add the pixel data ofthe nine pixels together including the target pixel and the surroundingeight pixels, or may add, to the pixel data of the target pixel, thepixel data of the surrounding eight pixels after being subjected toparticular weighting processing.

An example of the weighting processing performed on the pixel data ofthe surrounding pixels is explained below. As is apparent from FIG. 3 orFIG. 32 described below, R, G and B show the same sensitivity at thewavelength IR2 assigned to G light and the wavelength IR3 assigned to Blight.

Therefore, when the pixel of G or B is the target pixel as shown initems (b) to (d) of FIG. 13, and the surrounding pixel adding unit 521adds the pixel data of the surrounding eight pixels without weightingprocessing to the pixel data for G or B, the sensitivity is increased byapproximately 9 times.

On the other hand, R, G and B greatly differ in sensitivity at thewavelength IR1 assigned to R light, as is apparent from FIG. 3 or FIG.32 described below.

Thus, when the pixel of R is the target pixel as shown in item (a) ofFIG. 13, and the surrounding pixel adding unit 521 adds the pixel dataof the surrounding eight pixels without weighting processing to thepixel data for R, the sensitivity is increased, not by 9 times, but by asmaller multiple.

Therefore, the color balance is lost if the surrounding pixel addingunit 521 adds the pixel data of the surrounding eight pixels withoutweighting processing to the pixel data of the target pixel since the Rsignal and the G or B signal differ in sensitivity.

Accordingly, the surrounding pixel adding unit 521 preferably adds, tothe pixel data of the target pixel, the pixel data of the surroundingeight pixels after being subjected to the weighting processing. Thesurrounding pixel adding unit 521 preferably performs the weightingprocessing differently between the case where the pixel of R is thetarget pixel and the case where the pixel of G or B is the target pixel.The surrounding pixel adding unit 521 may perform the weightingprocessing differently between the case where the pixel of G is thetarget pixel and the case where the pixel of B is the target pixel.

FIG. 14 shows a first example of the weighting processing. The numbersindicated below the respective pixels of R, Gb, Gr and B shown in FIG.14 are weighting coefficients.

The ratio of the spectral sensitivities of R, G and B of the imagingunit 3 is set to R:G:B=1:0.8:0.5 at the wavelength IR1, and set toR:G:B=1:1:1 at the wavelengths IR2 and IR3. The basic weightingcoefficients are set to, for example, the target pixel: thevertical-horizontal pixels: the diagonal pixels=100:70:50.

In such a case, when the pixel of Gr, Gb or B is the target pixel asshown in FIG. 14, the pixel data of the surrounding eight pixels may bemultiplied by the basic weighting coefficients.

When the pixel of R is the target pixel, as the infrared light with thewavelength IR1 is projected, it is preferable to use the weightingcoefficients within the ratio of the spectral sensitivities of R, G andB described above. At the wavelength IR1, the spectral sensitivity ofR/the spectral sensitivity of G, which is 1/0.8, is 1.25, and thespectral sensitivity of R/the spectral sensitivity of B, which is 1/0.5,is 2.0.

Accordingly, when the pixel of R is the target pixel as shown in FIG.14, the weighting coefficient of 87, which is obtained by 70×1.25, ispreferably used for the vertical-horizontal pixel data for G, and theweighting coefficient of 100, which is obtained by 50×2.0, is preferablyused for the diagonal pixel data for B.

The ratio of the spectral sensitivities of R, G and B of the imagingunit 3 varies depending on the type of the imaging element 31 used or amanufacturer which manufactures the device. The weighting coefficientsare preferably determined depending on the type of the imaging element31 to be used.

FIG. 15 shows a second example of the weighting processing. The ratio ofthe spectral sensitivities of R, G and B of the imaging unit 3 is set toR:G:B=1:0.08:0.5 at the wavelength IR1, set to R:G:B=1:1:1: at thewavelength IR2, and set to R:G:B=1:0.95:1 at the wavelength IR3. Thebasic weighting coefficients are the same as those in the first example.

As shown in FIG. 15, when the pixel of Gr or Gb is the target pixel, thepixel data of the surrounding eight pixels may be multiplied by thebasic weighting coefficients.

When the pixel of R is the target pixel, it is preferable to use theweighting coefficients within the ratio of the spectral sensitivities ofR, G and B as in the case of the first example. As shown in FIG. 15,when the pixel of R is the target pixel, the weighting coefficients arethe same as those in the first example.

When the pixel of B is the target pixel, as the infrared light with thewavelength IR3 is projected, it is preferable to use the weightingcoefficients within the ratio of the spectral sensitivities of R, G andB described above. At the wavelength IR3, the spectral sensitivity ofB/the spectral sensitivity of G, which is 1/0.95, is 1.05.

Accordingly, when the pixel of B is the target pixel as shown in FIG.15, the weighting coefficient of 73, which is obtained by 70×1.05, ispreferably used for the vertical-horizontal pixel data for G. Theweighting coefficient for the pixel data for R may be the basicweighting coefficient of 50.

Some of the weighting coefficients for the pixel data of the surroundingeight pixels may be zero. In other words, the surrounding pixel addingunit 521 does not necessarily add all of the pixel data of thesurrounding eight pixels to the pixel data of the target pixel.

For example, the surrounding pixel adding unit 521 does not necessarilyadd the pixel data of the diagonal pixels but may only add the pixeldata of the vertical and horizontal pixels to the pixel data of thetarget pixel.

FIG. 16 shows a third example of the weighting processing, in which theweighting coefficient for the diagonal pixels is zero. The basicweighting coefficients are set to, for example, the target pixel:thevertical-horizontal pixels:the diagonal pixels=100:50:0.

The ratio of the spectral sensitivities of R, G and B of the imagingunit 3 is conceived to be the same as that in the first example. Asshown in FIG. 16, when the pixel of R is the target pixel, the weightingcoefficient of 62, which is obtained by 50×1.25, is preferably used forthe vertical-horizontal pixel data for G.

The values of the weighting coefficients shown in FIG. 14 to FIG. 16 aremerely examples for ease of recognition of the weighting processingperformed on the surrounding pixels. Actual weighting coefficients maybe determined as appropriate in view of a dynamic range of image signalsoutput from the image output unit 6 or the like.

Here, there is a known imaging element capable of reading out aplurality of pixels as a single pixel, which is called binning. When theimaging element possessing the binning function is used as the imagingelement 31, the adding processing may be performed not by thesurrounding pixel adding unit 521 but by the imaging element with thisbinning function. The binning processing by the imaging element issubstantially equivalent to the adding processing by the surroundingpixel adding unit 521.

The frame FmIR1, the frame FmIR3, and the frame FmIR2 shown in items (a)to (c) of FIG. 17 are the same as the frame FmIR1, the frame FmIR3, andthe frame FmIR2 shown in items (a) to (c) of FIG. 9, respectively. Initems (d) to (f) of FIG. 17, each of added pixel data R1 ad, Gr1 ad, Gb1ad, B1 ad, R2 ad, Gr2 ad, Gb2 ad, B2 ad, R3 ad, Gr3 ad, Gb3 ad and B3 adis obtained in a manner such that the pixel data of the surroundingeight pixels are added to the pixel data for each of R, Gr, Gb, and B.

The surrounding pixel adding unit 521 subjects the pixel data in each ofthe frames FmIR1, FmIR3 and FmIR2 to adding processing shown in FIG. 13so as to generate frame FmIR1 ad, frame FmIR2 ad and frame FmIR3 adshown in items (d) to (f) of FIG. 17.

The frames FmIR1 ad, FmIR2 ad and FmIR3 ad shown in items (a) to (c) ofFIG. 18 are the same as the frames FmIR1 ad, FmIR2 ad and FmIR3 ad shownin items (d) to (f) of FIG. 17, respectively.

As in the case of the first intermediate mode, the same-position pixeladding unit 522 adds, to the pixel data R1 ad located in the frame FmIR1ad, the pixel data R2 ad and R3 ad located at the same pixel positionsin the respective frames FmIR2 ad and FmIR3 ad so as to generate addedpixel data R123 ad according to the formula (1).

The same-position pixel adding unit 522 adds, to the pixel data Gr2 ad,Gb2 ad located in the frame FmIR2 ad, the pixel data Gr1 ad, Gb1 ad, Gr3ad, and Gb3 ad located at the same pixel positions in the respectiveframes FmIR1 ad and FmIR3 ad so as to generate added pixel data Gr123 adand Gb123 ad according to the formula (2).

The same-position pixel adding unit 522 adds, to the pixel data B3 adlocated in the frame FmIR3 ad, the pixel data B1 ad and B2 ad located atthe same pixel positions in the respective frames FmIR1 ad and FmIR2 adso as to generate added pixel data B123 ad according to the formula (3).

As in the case of the first intermediate mode, the synthesizing unit 523selects the added pixel data R123 ad in the frame FmIR1 ad, the addedpixel data Gr123 ad and Gb123 ad in the frame FmIR2 ad and the addedpixel data B123 ad in FmIR3 ad and synthesizes the respective addedpixel data. The synthesizing unit 523 thus generates frame FmIR123 ad ofthe synthesized image signals shown in item (d) of FIG. 18.

The synthesizing unit 523 generates the frame FmIR123 ad in which therespective added pixel data R123 ad, Gr123 ad, Gb123 ad and B123 ad arearranged so as to have the same array as the filter elements in thecolor filter 32.

Item (a) of FIG. 19 shows the frame FmIR123 ad. The demosaicing unit 54computes pixel data for R for pixel positions where no pixel data for Ris present by use of the surrounding pixel data for R so as to generateinterpolated pixel data R123 adi for R. The demosaicing unit 54generates R frame FmIR123 adR in which all pixels in one frame shown initem (b) of FIG. 19 are composed of the pixel data for R.

The demosaicing unit 54 computes pixel data for G for pixel positionswhere no pixel data for G is present by use of the surrounding pixeldata for G so as to generate interpolated pixel data G123 adi for G. Thedemosaicing unit 54 generates G frame FmIR123 adG in which all pixels inone frame shown in item (c) of FIG. 19 are composed of the pixel datafor G.

The demosaicing unit 54 computes pixel data for B for pixel positionswhere no pixel data for B is present by use of the surrounding pixeldata for B so as to generate interpolated pixel data B123 adi for B. Thedemosaicing unit 54 generates B frame FmIR123 adB in which all pixels inone frame shown in item (d) of FIG. 19 are composed of the pixel datafor B.

The first intermediate mode and the first night-vision mode differ fromeach other in that the surrounding pixel adding unit 521 is inactive inthe first intermediate mode, and the surrounding pixel adding unit 521is active in the first night-vision mode. The mode switching unit 72 isonly required to activate the surrounding pixel adding unit 521 when inthe night-vision mode.

The operation of the demosaicing unit 54 in the night-vision mode issubstantially the same as that in the normal mode and in theintermediate mode. The normal mode, the intermediate mode and thenight-vision mode may share the signal processing unit such as thedemosaicing unit 54 in the image processing unit 5.

<Night-Vision Mode: Second Night-Vision Mode>

Operations in the second night-vision mode are explained below withreference to FIG. 20 and FIG. 21. Note that the same operations as thosein the first night-vision mode are not repeated in the secondnight-vision mode. Here, the frames FmIR1 ad, FmIR2 ad and FmIR3 adshown in items (a) to (c) in FIG. 20 are the same as the frames FmIR1ad, FmIR2 ad and FmIR3 ad shown in items (a) to (c) in FIG. 18.

The synthesizing unit 523 selects pixel data R1 ad for R in the frameFmIR1 ad, pixel data Gr2 ad and Gb2 ad for G in the frame FmIR2 ad, andpixel data B3 ad for B in FmIR3 ad and synthesizes the respective pixeldata. The synthesizing unit 523 thus generates frame FmIR123′ad of thesynthesized image signals shown in item (d) of FIG. 20.

The synthesizing unit 523 generates the frame FmIR123′ad in which therespective pixel data R1 ad, Gr2 ad, Gb2 ad and B3 ad are arranged so asto have the same array as the filter elements in the color filter 32.

As explained with reference to FIG. 13, the pixel data R1 ad for red inthe frame FmIR123′ad is generated from the pixel data obtained from awider region than the region used for generating the pixel data for redwhen in the intermediate mode.

The pixel data Gr2 ad for green in the frame FmIR123′ad is generatedfrom the pixel data obtained from a wider region than the region usedfor generating the pixel data for green when in the intermediate mode.

The pixel data B3 ad for blue in the frame FmIR123′ad is generated fromthe pixel data obtained from a wider region than the region used forgenerating the pixel data for blue when in the intermediate mode.

As in the case of the second intermediate mode, the same-position pixeladding unit 522 in the second night-vision mode defines the coefficientKa in the formula (1) as 1 and the other coefficients Kb and Kc as 0,defines the coefficient ke in the formula (2) as 1 and the othercoefficients kd and kf as 0, and defines the coefficient ki in theformula (3) as 1 and the other coefficients kg and kh as 0.

Therefore, the value of the pixel data R1 ad in the frame FmIR1 ad, thevalues of the pixel data Gr2 ad and Gb2 ad in the frame FmIR2 ad and thevalue of the pixel data B3 ad in the frame FmIR3 ad each remain as is.

Accordingly, the synthesizing unit 523 can generate the frame FmIR123′adby selecting the pixel data R1 ad in the frame FmIR1 ad, the pixel dataGr2 ad and Gb2 ad in the frame FmIR2 ad and pixel data B3 ad in theframe FmIR3 ad, in the same manner as the operations in the firstnight-vision mode.

The demosaicing in the demosaicing unit 54 is explained below withreference to FIG. 21. Item (a) of FIG. 21 shows the frame FmIR123′ad.The demosaicing unit 54 computes pixel data for R for pixel positionswhere no pixel data for R is present by use of the surrounding pixeldata R1 ad so as to generate interpolated pixel data R1 adi for R. Thedemosaicing unit 54 generates R frame FmIR123′adR in which all pixels inone frame shown in item (b) of FIG. 21 are composed of the pixel datafor R.

The demosaicing unit 54 computes pixel data for G for pixel positionswhere no pixel data for G is present by use of the surrounding pixeldata Gr2 ad and Gb2 ad so as to generate interpolated pixel data G2 adifor G. The demosaicing unit 54 generates G frame FmIR123′adG in whichall pixels in one frame shown in item (c) of FIG. 21 are composed of thepixel data for G.

The demosaicing unit 54 computes pixel data for B for pixel positionswhere no pixel data for B is present by use of the surrounding pixeldata B3 ad so as to generate interpolated pixel data B3 adi for B. Thedemosaicing unit 54 generates B frame FmIR123′adB in which all pixels inone frame shown in item (d) of FIG. 21 are composed of the pixel datafor B.

The second intermediate mode and the second night-vision mode differfrom each other in that the surrounding pixel adding unit 521 isinactive in the second intermediate mode, and the surrounding pixeladding unit 521 is active in the second night-vision mode.

While the pixel data for each color is generated from the pixel dataobtained from the region corresponding to each color filter in the lightreceiving elements in the intermediate mode, the pixel data for eachcolor is generated, in the night-vision mode, from the pixel dataobtained from a wider region than the region used for generating thepixel data for each color in the intermediate mode, as the surroundingpixels are added in the night-vision mode.

<Example of Mode Switch>

An example of the mode switch by the mode switching unit 72 is explainedbelow with reference to FIG. 22. Item (a) of FIG. 22 is an exampleschematically showing a state of change in environmental brightness withthe passage of time from daytime to nighttime.

As shown in item (a) of FIG. 22, the brightness decreases gradually withthe passage of time from daytime to nighttime and results in almosttotal darkness after time t3. Item (a) of FIG. 22 shows the brightnessrepresenting the substantial amount of visible light and indicates thatalmost no visible light is present after time t3.

The controller 7 can determine the environmental brightness based on abrightness level of image signals (image data) input from the imageprocessing unit 5. As shown item (b) of FIG. 22, the mode switching unit72 selects the normal mode when the brightness is predeterminedthreshold Th1 (first threshold) or greater, selects the intermediatemode when the brightness is less than the threshold Th1 andpredetermined threshold Th2 (second threshold) or greater, and selectsthe night-vision mode when the brightness is less than the thresholdTh2.

The imaging device according to the present embodiment automaticallyswitches the modes in such a manner as to select the normal mode by timet1 at which the brightness reaches the threshold Th1, select theintermediate mode in the period from time t1 to time t2 at which thebrightness reaches the threshold Th2, and select the night-vision modeafter time t2. In item (b) of FIG. 22, the intermediate mode may beeither the first intermediate mode or the second intermediate mode, andthe night-vision mode may be either the first night-vision mode or thesecond night-vision mode.

Although the brightness immediately before time t3 at which almost novisible light remains is defined as the threshold Th2 in item (a) ofFIG. 22, the brightness at time t3 may be determined as the thresholdTh2.

As shown in item (c) of FIG. 22, the mode switching unit 72 may dividethe intermediate mode into two periods: a first half period toward timet1 as the first intermediate mode in which the amount of visible lightis relatively high; and a second half period toward time t2 as thesecond intermediate mode in which the amount of visible light isrelatively low. In item (c) of FIG. 22, the night-vision mode may beeither the first night-vision mode or the second night-vision mode.

In the imaging device according to the present embodiment, theprojection controller 71 controls the ON/OFF state of the infraredprojector 9, and the mode switching unit 72 switches the respectivemembers in the image processing unit 5 between the active state and theinactive state, so as to implement the respective modes.

As shown in FIG. 23, the normal mode is a state where the infraredprojector 9 is turned OFF, the surrounding pixel adding unit 521, thesame-position pixel adding unit 522 and the synthesizing unit 523 areinactive, and the demosaicing unit 54 is active.

The first intermediate mode is implemented in a state where the infraredprojector 9 is turned ON, the surrounding pixel adding unit 521 isinactive, and the same-position pixel adding unit 522, the synthesizingunit 523 and the demosaicing unit 54 are active. The second intermediatemode is implemented in a state where the infrared projector 9 is turnedON, the surrounding pixel adding unit 521 and the same-position pixeladding unit 522 are inactive, and the synthesizing unit 523 and thedemosaicing unit 54 are active.

The same-position pixel adding unit 522 can be easily switched betweenthe active state and the inactive state by appropriately setting thecoefficients ka to ki in the formulae (1) to (3) as described above.

The first night-vision mode is implemented in a state where the infraredprojector 9 is turned ON, and the surrounding pixel adding unit 521, thesame-position pixel adding unit 522, the synthesizing unit 523 and thedemosaicing unit 54 are all active. The second night-vision mode isimplemented in a state where the infrared projector 9 is turned ON, thesame-position pixel adding unit 522 is inactive, and the surroundingpixel adding unit 521, the synthesizing unit 523 and the demosaicingunit 54 are active.

Here, the surrounding pixel adding unit 521 can be activated for theprocessing of adding the surrounding pixels by setting the coefficientto more than 0 (for example, 1) by which the surrounding pixel data ismultiplied in the calculation formula for adding the surrounding pixeldata to the pixel data of the target pixel.

The surrounding pixel adding unit 521 can be inactivated for theprocessing of adding the surrounding pixels by setting the coefficientto 0 by which the surrounding pixel data is multiplied in thecalculation formula.

The surrounding pixel adding unit 521 thus can also be easily switchedbetween the active state and the inactive state by setting thecoefficients as appropriate.

<First Modified Example of Imaging Device>

The method of detecting the environmental brightness by the controller 7is not limited to the method based on the brightness level of the imagesignals.

As shown in FIG. 24, the environmental brightness may be detected by abrightness sensor 11. In FIG. 24, the environmental brightness may bedetermined based on both the brightness level of the image signals andthe environmental brightness detected by the brightness sensor 11.

<Second Modified Example of Imaging Device>

The controller 7 may briefly estimate the environmental brightness basedon the season (date) and the time (time zone) during a year, instead ofthe direct detection of the environmental brightness, so that the modeswitching unit 72 switches the modes.

As shown in FIG. 25, the normal mode, the intermediate mode and thenight-vision mode are set in a mode setting table 12 depending on thecombination of the date and time. A time clock 73 in the controller 7manages the date and the time. The controller 7 refers to the date andthe time indicated on the time clock 73 to read out the mode set in themode setting table 12.

The projection controller 71 and the mode switching unit 72 control theimaging device so that the mode read from the mode setting table 12 isselected.

<Third Modified Example of Imaging Device>

As shown in FIG. 26, a user may control the imaging device with anoperation unit 13 by manually selecting one of the modes so as to setthe projection controller 71 and the mode switching unit 72 to theselected mode. The operation unit 13 may be operated using operationbuttons provided on the casing of the imaging device or by a remotecontroller.

<Image Signal Processing Method Regarding Mode Switch>

The image signal processing method regarding the mode switch executed bythe imaging device shown in FIG. 1 is again explained with reference toFIG. 27.

In FIG. 27, once the imaging device starts operating, the controller 7determines whether the environmental brightness is the threshold Th1 orgreater in step S1. When the environmental brightness is the thresholdTh1 or greater (YES), the controller 7 executes the processing in thenormal mode in step S3. When the environmental brightness is not thethreshold Th1 or greater (NO), the controller 7 determines whether theenvironmental brightness is threshold Th2 or greater in step S2.

When the environmental brightness is the threshold Th2 or greater (YES),the controller 7 executes the processing in the intermediate mode instep S4. When the environmental brightness is not the threshold Th2 orgreater (NO), the controller 7 executes the processing in thenight-vision mode in step S5.

The controller 7 returns to the processing in step S1 after executingthe processing from steps S3 to S5 and repeats the respective followingsteps.

FIG. 28 shows the specific processing in the normal mode in step S3. InFIG. 28, the controller 7 (the projection controller 71) turns off theinfrared projector 9 in step S31. The controller 7 inserts the infraredcut filter 21 in step S32. The controller 7 (the mode switching unit 72)connects the switches 51 and 53 to the respective terminals Tb in stepS33. The execution order from steps S31 to S33 is optional. The stepsS31 to S33 can be executed simultaneously.

The controller 7 directs the imaging unit 3 to image an object in stepS34. The controller 7 controls the image processing unit 5 in step S35so that the demosaicing unit 54 subjects, to demosaicing, a framecomposing image signals generated when the imaging unit 3 images theobject.

FIG. 29 shows the specific processing in the intermediate mode in stepS4. In FIG. 29, the controller 7 (the projection controller 71) turns onthe infrared projector 9 in step S41 so that the projecting portions 91to 93 project infrared light with the respective wavelengths IR1 to IR3in a time division manner.

The controller 7 inserts the dummy glass 22 instep S42. The controller 7(the mode switching unit 72) connects the switches 51 and 53 to therespective terminals Ta in step S43. The execution order from steps S41to S43 is optional. The steps S41 to S43 may be executed simultaneously.

The controller 7 directs the imaging unit 3 to image an object in stepS44. The imaging unit 3 images the object in a state where the infraredlight with the wavelength IR1 assigned to R, the infrared light with thewavelength IR2 assigned to G and the infrared light with the wavelengthIR3 assigned to B, are each projected.

The controller 7 (the mode switching unit 72) controls the pre-signalprocessing unit 52 in step S45 so as to inactivate the surrounding pixeladding unit 521 and activate the synthesizing unit 523 to generatesynthesized image signals.

The respective frames composing the image signals generated when theimaging unit 3 images the object in the state where the infrared lightwith the respective wavelengths IR1, IR2 and IR3 is projected, aredefined as a first frame, a second frame, and a third frame.

The synthesizing unit 523 arranges the pixel data for the three primarycolors based on the pixel data for R in the first frame, the pixel datafor G in the second frame and the pixel data for B in the third frame soas to have the same array as the filter elements in the color filter 32.The synthesizing unit 523 thus generates the synthesized image signalsin a manner such that the image signals in the first to third frames aresynthesized in one frame.

The controller 7 controls the image processing unit 5 in step S46 sothat the demosaicing unit 54 subjects the frame composing thesynthesized image signals to demosaicing.

The demosaicing unit 54 executes, based on the frame of the synthesizedimage signals, demosaicing for generating an R frame, a G frame and a Bframe so as to sequentially generate the frames of the three primarycolors subjected to demosaicing.

The demosaicing unit 54 can generate the R frame by interpolating thepixel data for R in the pixel positions where no pixel data for R ispresent. The demosaicing unit 54 can generate the G frame byinterpolating the pixel data for G in the pixel positions where no pixeldata for G is present. The demosaicing unit 54 can generate the B frameby interpolating the pixel data for B in the pixel positions where nopixel data for B is present.

When executing the operations in the first intermediate mode, thecontroller 7 activates the same-position pixel adding unit 522 in stepS45. When executing the operations in the second intermediate mode, thecontroller 7 inactivates the same-position pixel adding unit 522 in stepS45.

FIG. 30 shows the specific processing in the night-vision mode in stepS5. In FIG. 30, the controller 7 (the projection controller 71) turns onthe infrared projector 9 in step S51 so that the projecting portions 91to 93 project infrared light with the respective wavelengths IR1 to IR3in a time division manner.

The controller 7 inserts the dummy glass 22 in step S52. The controller7 (the mode switching unit 72) connects the switches 51 and 53 to therespective terminals Ta in step S53. The execution order from steps S51to S53 is optional. The steps S51 to S53 may be executed simultaneously.

The controller 7 directs the imaging unit 3 to image an object in stepS54. The controller 7 (the mode switching unit 72) controls thepre-signal processing unit 52 in step S55 so as to activate thesurrounding pixel adding unit 521 and the synthesizing unit 523 togenerate synthesized image signals.

The controller 7 controls the image processing unit 5 in step S56 sothat the demosaicing unit 54 subjects the frame composing thesynthesized image signals to demosaicing.

When executing the operations in the first night-vision mode, thecontroller 7 activates the same-position pixel adding unit 522 in stepS55. When executing the operations in the second night-vision mode, thecontroller 7 inactivates the same-position pixel adding unit 522 in stepS55.

<Image Signal Processing Program Regarding Mode Switch>

In FIG. 1, the controller 7 or the integrated portion of the imageprocessing unit 5 and the controller 7 may be composed of a computer(microcomputer), and an image signal processing program (computerprogram) may be executed by the computer so as to implement the sameoperations as those in the imaging device described above. Theintegrated portion may further include the image output unit 6 so as tobe composed of the computer.

An example of a procedure of the processing executed by the computerwhen the processing in the intermediate mode executed in step S4 shownin FIG. 27 is included in the image signal processing program, isexplained below with reference to FIG. 31. FIG. 31 shows the processingexecuted by the computer directed by the image signal processing programregarding the mode switch.

In FIG. 31, the image signal processing program directs the computer tocontrol the infrared projector 9 in step S401 to project infrared lightwith the wavelengths IR1, IR2 and IR3 assigned to R, G and B,respectively.

The step in step S401 may be executed by an external unit outside of theimage signal processing program. In FIG. 31, the step of inserting thedummy glass 22 is omitted. The step of inserting the dummy glass 22 maybe executed by the external unit outside of the image signal processingprogram.

The image signal processing program directs the computer in step S402 toobtain the pixel data composing the first frame of the image signalsgenerated when the imaging unit 3 images the object in the state wherethe infrared light with the wavelength IR1 is projected.

The image signal processing program directs the computer in step S403 toobtain the pixel data composing the second frame of the image signalsgenerated when the imaging unit 3 images the object in the state wherethe infrared light with the wavelength IR2 is projected.

The image signal processing program directs the computer in step S404 toobtain the pixel data composing the third frame of the image signalsgenerated when the imaging unit 3 images the object in the state wherethe infrared light with the wavelength IR3 is projected. The executionorder from steps S402 to 404 is optional.

The image signal processing program directs the computer in step S405 toarrange the respective pixel data for R, G and B in such a manner as tohave the same array as the filter elements in the color filter 32 so asto generate the synthesized image signals synthesized in one frame.

In the intermediate mode, the image signal processing program does notdirect the computer to execute the processing of adding the surroundingpixels in step S405.

The image signal processing program directs the computer in step S406 tosubject the frame of the synthesized image signals to demosaicing so asto generate the frames of R, G and B.

Although not illustrated in the drawing, the image signal processingprogram may direct the computer to execute the processing of adding thesurrounding pixels in step S405 in FIG. 31 when the processing in thenight-vision mode executed in step S5 shown in FIG. 27 is included inthe image signal processing program.

The image signal processing program may be a computer program recordedin a readable storage medium. The image signal processing program may beprovided in a state of being stored in the storage medium, or may beprovided via a network such as Internet in a manner such that the imagesignal processing program is downloaded to the computer. The storagemedium readable on the computer may be an arbitrary non-transitorystorage medium such as CD-ROM and DVD-ROM.

The imaging device configured as shown in FIG. 1 may include two or moresets of the respective members as necessary so as to execute theintermediate mode and the night-vision mode simultaneously. In such acase, the image output unit 6 may output both the image signalsgenerated in the intermediate mode and the image signals generated inthe night-vision mode.

The mode switching unit 72 may switch between a state where the imageoutput unit 6 outputs the image signals generated in the intermediatemode and a state where the image output unit 6 outputs the image signalsgenerated in the night-vision mode. In such a case, the mode switchingunit 72 may switch the states depending on the environmental brightnessor the time, as described above. In addition, the image processing unit5 may be provided separately from the other members.

Further, the normal mode may be switched directly to the night-visionmode, or the night-vision mode may be switched directly to the normalmode, bypassing the intermediate mode.

When the intermediate mode is not present, either the normal mode or thenight-vision mode may be selected and used even under the condition thatthe intermediate mode is appropriate. In such a case, although finecolor image signals are not obtained as in the case of using theintermediate mode, images can be captured.

Even the imaging device only equipped with the normal mode and thenight-vision mode can image objects in variable conditions ofenvironmental brightness, such as a case where a surveillance cameracaptures objects throughout the day.

Further, the normal mode may be switched to the intermediate mode, andthe intermediate mode may be switched to the normal mode, instead ofusing the night-vision mode. If the night-vision mode is constantlyinactive, the night-vision mode may be eliminated from the imagingdevice.

The night-vision mode is not necessarily used in an area where, forexample, electric lighting is equipped. The imaging device only equippedwith the normal mode and the intermediate mode may be used in the casewhere the night-vision mode is not necessarily used.

When the night-vision mode is not present, the intermediate mode may beused even under the condition that the night-vision mode is appropriate.In such a case, although fine color image signals are not obtained as inthe case of using the night-vision mode, images can be captured.

Even the imaging device only equipped with the normal mode and theintermediate mode can image objects in variable conditions ofenvironmental brightness, as in the case described above.

<Method of Determining Relationship Between Visible Light Amount andInfrared Light Amount>

As described above, the relationship between the amount of visible lightand the amount of infrared light varies depending on the surroundingenvironment. If the white balance of the image signals is adjusted basedon the state where the amount of visible light and the amount ofinfrared light has a specific relationship, the white balance is lostonce the amount of visible light and the amount of infrared lightdeviate from the specific relationship. In other words, if therelationship between the respective light amounts changes, the color ofan object cannot be reproduced in high definition under the respectiveconditions.

In view of this, the determination unit 78 analyzes thesuperior-subordinate relationship between the respective light amountsaccording to a first example or a second example of a determinationmethod described below. First, the spectral sensitive characteristics inthe imaging unit 3 described in FIG. 3 are considered with reference toFIG. 32.

As described above, the wavelengths IR1, IR2 and IR3 are set to 780 nm,940 nm and 870 nm, respectively. As shown in FIG. 32, R sensitivity inthe wavelength IR1 assigned to R is defined as Str, G sensitivity in thewavelength IR2 assigned to G is defined as Stg, and B sensitivity in thewavelength IR3 assigned to B is defined as Stb.

As is apparent from FIG. 32, the values of Str, Stg and Stb greatlydiffer from each other. Therefore, when the emission power is constantin the infrared light with the respective wavelengths IR1, IR2 and IR3,the exposure amount of the imaging unit 3 varies depending on whichinfrared light is projected among the infrared light with thewavelengths IR1, IR2 and IR3.

<First Example of Determination Method>

The first example of the determination method is explained below withreference to FIG. 33 to FIG. 36 and FIG. 38. The infrared light with thewavelengths IR1 to IR3 is conceived to have the same emission power. Asthe first example, the determination unit 78 analyzes thesuperior-subordinate relationship between the respective light amountsby comparing the exposure amounts of the imaging unit 3 at least in twoperiods among the exposure amounts in the state where infrared lightwith the respective wavelengths IR1, IR2 and IR3 is selectivelyprojected in each of predetermined periods.

FIG. 33 shows a state where the amount of visible light is sufficientlygreater than the amount of infrared light and thus the visible light isthe predominant light. Items (a) and (b) of FIG. 33 partly show items(a) and (b) of FIG. 8, respectively. The infrared projector 9selectively projects the infrared light with the respective wavelengthsIR1 to IR3 as shown in item (a) of FIG. 33, and the imaging unit 3 emitsthe light as shown in item (b) of FIG. 33.

The exposure amount in the period in which the infrared light with thewavelength IR1 is projected is defined as AepR, the exposure amount inthe period in which the infrared light with the wavelength IR2 isprojected is defined as AepG, and the exposure amount in the period inwhich the infrared light with the wavelength IR3 is projected is definedas AepB.

When the visible light is the predominant light, the exposure amounts ofthe imaging unit 3 are hardly influenced by the infrared light projectedfrom the infrared projector 9. Therefore, as shown in item (c) of FIG.33, the exposure amounts AepR, AepG and AepB of the imaging unit 3 aresubstantially the same, whether the infrared light projected has thewavelength IR1, IR2 or IR3. The exposure amounts AepR, AepG and AepB ofthe imaging unit 3 are substantially the same in the state where thevisible light is the predominant light.

FIG. 34 shows a state where the amount of infrared light is sufficientlygreater than the amount of visible light and thus the infrared light isthe predominant light. Items (a) and (b) of FIG. 34 are the same asitems (a) and (b) of FIG. 33, respectively. When the infrared light isthe predominant light, the exposure amounts of the imaging unit 3 areinfluenced by the infrared light projected from the infrared projector9.

Therefore, as shown in item (c) of FIG. 34, the exposure amounts AepR,AepG and AepB of the imaging unit 3 differ from each other depending onwhich infrared light is projected among the infrared light with thewavelengths IR1, IR2 and IR3. The exposure amounts AepR, AepG and AepBfulfill the relationship of AepR>AepB>AepG, which is reflective of thedifference among the sensitivities Str, Stg and Stb described in FIG.32.

As is apparent from the comparison between item (c) of FIG. 33 and item(c) of FIG. 34, the determination unit 78 determines that the visiblelight is the predominant light when the exposure amounts of the imagingunit 3 are approximately uniform in the three-divided periods in oneframe, and that the infrared light is the predominant light when theexposure amounts of the imaging unit 3 differ in the three-dividedperiods.

More particularly, it can be determined that the visible light is thepredominant light when at least two of the exposure amounts AepR, AepGand AepB are compared, and the difference thereof is a predeterminedthreshold or less, and that the infrared light is the predominant lightwhen the difference thereof exceeds the predetermined threshold. Thedetermination accuracy can be improved by comparing the three exposureamounts AepR, AepG and AepB.

In case of the comparison between two of the exposure amounts AepR, AepGand AepB, the exposure amount AspR and the exposure amount AspG, ofwhich difference is the largest, are preferably compared.

The specific configuration and operation of the determination unit 78implementing the first example of the determination method are explainedin more detail below with reference to FIG. 35. FIG. 35 is a schematicview for explaining how to analyze the superior-subordinate relationshipof the light amounts by comparing the exposure amount AspR with theexposure amount AspG.

Item (a) of FIG. 35 is the arbitrary frame FmIR1 of the image datagenerated at the point where the infrared light with the wavelength IR1is being projected, as in the case of item (a) of FIG. 9. Item (b) ofFIG. 35 is the arbitrary frame FmIR2 of the image data generated at thepoint where the infrared light with the wavelength IR2 is beingprojected, as in the case of item (b) of FIG. 9. Note that the pixeldata Gr, Gb in FIG. 9 is indicated by G in FIG. 35 to which no index isadded.

Item (c) of FIG. 35 is a block diagram schematically showing thedetermination unit 78 in the first example. As shown in item (c) of FIG.35, the determination unit 78 includes averaging units 781, a differencecalculation unit 782, and a comparing unit 783.

The two averaging units 781 separately show the averaging unit 781 usedat the point where the pixel data in the frame FmIR1 is input and theaveraging unit 781 used at the point where the pixel data in the frameFmIR2 is input.

The averaging unit 781 averages the pixel data within a predeterminedregion in the frame FmIR1 to generate average Fave (IR1). The averagingunit 781 averages the pixel data within a predetermined region in theframe FmIR2 to generate average Fave (IR2). The averages Fave (IR1) andFave (IR2) are input to the difference calculation unit 782.

The predetermined region may be the entire frame, or may be part of theframe such as a central portion excluding the edge portions. Thepredetermined region may be plural frames. The predetermined region isconceived to be one frame in the following explanation.

The difference calculation unit 782 calculates the difference betweenthe average Fave (IR1) and the average Fave (IR2) to generate differenceVdf. Although not illustrated in the drawing, the difference Vdf isabsolutized by an absolute circuit and input into the comparing unit783.

The comparing unit 783 compares the difference Vdf and threshold Th10 togenerate determination data Ddet1, indicating that the visible light isthe predominant light when the difference Vdf is equal to or less thanthe threshold Th10, and that the infrared light is the predominant lightwhen the difference Vdf exceeds the threshold Th10.

The determination data Ddet1 may only include two values: “0” indicatedwhen visible light is the predominant light; and “1” indicated wheninfrared light is the predominant light. The determination data Ddet1 isinput into the color gain controller 79. Here, the determination dataDdet1 may be a value which varies depending on the difference Vdf so asto change the respective color gains depending on the value.

The operation of the determination unit 78 is explained in more detailbelow with reference to the flowchart shown in FIG. 36. In FIG. 36, thedetermination unit 78 determines whether the frame FmIR1 should begenerated now in step S101. When determining that the frame FmIR1 shouldbe generated now (YES), the determination unit 78 averages the pixeldata in the frame FmIR1 so as to generate the average Fave (IR1) in stepS102.

When determining that the frame FmIR1 should not be generated yet (NO),the determination unit 78 then determines in step S103 whether the frameFmIR2 should be generated instead. When determining that the frame FmIR2should be generated now (YES), the determination unit 78 averages thepixel data in the frame FmIR2 so as to generate the average Fave (IR2)in step S104. When determining that the frame FmIR2 should not begenerated yet (NO), the determination unit 78 returns to the processingin step S101.

The determination unit 78 calculates the difference between the averageFave (IR1) and the average Fave (IR2) to generate the difference Vdf instep S105. The determination unit 78 determines whether the differenceVdf is equal to or less than the threshold Th10 in step S106.

When the difference Vdf is equal to or less than the threshold Th10(YES), the determination unit 78 determines that the visible light isthe predominant light in step S107 and returns to the processing in stepS101. When the difference Vdf is not equal to or less than the thresholdTh10 (NO), the determination unit 78 determines that the infrared lightis the predominant light in step S108 and returns to the processing instep S101. The same procedure is repeated subsequently.

In the first example of the determination method described above, thesignal levels of the image signals (image data) input into thecontroller 7, particularly the averages of the pixel signals (pixeldata) in the respective frames, are compared so that the exposure amountAepR and the exposure amount AepG are substantially compared.

As shown in FIG. 37, the color gain controller 79 determines whether thevisible light is the predominant light based on the input determinationdata Ddet1 in step S201. When the visible light is the predominant light(YES), the color gain controller 79 controls the color gain setting unit62 to select the first set of color gains in step S202 and returns tothe processing in step S201.

When the visible light is not the predominant light (NO), the color gaincontroller 79 controls the color gain setting unit 62 to select thesecond set of color gains in step S203 and returns to the processing instep S201. The same procedure is subsequently repeated.

FIG. 34 and FIG. 35 each show the case where the emission power isconstant in the infrared light with the respective wavelengths IR1 toIR3. Alternatively, the superior-subordinate relationship of the lightamounts may be analyzed based on the exposure amounts AepR, AepG andAepB in such a manner as to intentionally vary the emission power ineach case of the infrared light with the wavelengths IR1 to IR3.

FIG. 38 shows a state where infrared light is the predominant light, asin the case of FIG. 34. Items (a) and (c) of FIG. 38 are the same asitems (a) and (b) of FIG. 34, respectively. As shown in item (b) of FIG.38, for example, the emission power of infrared light with thewavelength IR3 is assumed to be greater than the emission power ofinfrared light with the wavelengths IR1 and IR2.

Item (d) of FIG. 38 shows the exposure amounts AepR, AepG and AepB inthis case. The determination unit 78 may analyze thesuperior-subordinate relationship of the light amounts by comparing theexposure amount AepR with the exposure amount AepB, or may analyze thesuperior-subordinate relationship of the light amounts by comparing theexposure amount AepG with the exposure amount AepB. The determinationunit 78 may recognize the presence or absence of a particular pattern ofthe large-small relationship among the exposure amounts AepR, AepG andAepB so as to determine that visible light is the predominant light whenthere is no particular pattern and determine that infrared light is thepredominant light when the particular pattern is confirmed.

<Second Example of Determination Method>

The second example of the determination method is explained below withreference to FIG. 39 to FIG. 40. The infrared light with the respectivewavelengths IR1 to IR3 is conceived to have the same emission power. Asthe second example, the determination unit 78 analyzes thesuperior-subordinate relationship of the light amounts by using pixelsignals based on imaging signals generated in the state where theinfrared light with the wavelength IR1 is projected and pixel signalsbased on imaging signals generated in the state where other infraredlight is projected.

The determination unit 78 uses pixel signals of R and pixel signals of Gor B. The other infrared light may be either the infrared light with thewavelength IR2 or the infrared light with the wavelength IR3.

As shown in FIG. 32, the sensitivity Str for R greatly differs from thesensitivities for G and B at the wavelength IR1. The difference betweenthe sensitivity Str for R and the sensitivity Stb1 for B is particularlylarge. The sensitivities Str2 and Stb2 for R and B at the wavelength IR2are substantially the same as the sensitivity Stg for G, and there isalmost no difference therebetween. The difference between thesensitivity Stb for B and the sensitivities for R and G is alsoinsignificant at the wavelength IR3.

In the second example of the determination method, thesuperior-subordinate relationship of the light amounts is analyzed byuse of the difference among the sensitivities for R, G and B at therespective wavelengths IR1 to IR3.

Since the difference between the sensitivity Str for R and thesensitivity Stb1 for B is the largest at the wavelength IR1, the pixelsignals of R are preferably compared with the pixel signals of B ratherthan compared with the pixel signals of G. Therefore, in this example,the determination unit 78 uses the pixel signals of R and the pixelsignals of B in the state where the infrared light with the wavelengthIR1 is projected.

Although the determination unit 78 may use either the pixel signals of Rand B in the state where the infrared light with the wavelength IR2 isprojected or the pixel signals of R and B in the state where theinfrared light with the wavelength IR3 is projected, in this situationthe determination unit 78 uses the former.

The specific configuration and operation of the determination unit 78implementing the second example of the determination method areexplained below with reference to FIG. 39. FIG. 39 is a schematic viewfor explaining how to analyze the superior-subordinate relationship ofthe light amounts, as in the case of FIG. 35.

Item (a) of FIG. 39 is the arbitrary frame FmIR1 of the image datagenerated at the point where the infrared light with the wavelength IR1is being projected, as in the case of item (a) of FIG. 35. Item (b) ofFIG. 39 is the arbitrary frame FmIR2 of the image data generated at thepoint where the infrared light with the wavelength IR2 is beingprojected, as in the case of item (b) of FIG. 35.

Item (c) of FIG. 39 schematically shows a state where pixels for R inthe frame FmIR1 are only extracted. Item (d) of FIG. 39 schematicallyshows a state where pixels for B in the frame FmIR1 are only extracted.

Item (e) of FIG. 39 schematically shows a state where pixels for R inthe frame FmIR2 are only extracted. Item (f) of FIG. 39 schematicallyshows a state where pixels for B in the frame FmIR2 are only extracted.

Item (g) of FIG. 39 is a block diagram schematically showing thedetermination unit 78 in the second example. As shown in item (g) ofFIG. 39, the determination unit 78 includes averaging units 784 and 785,difference calculation units 786 and 787, and a comparing unit 788.

The respective two averaging units 784 and 785 separately show theaveraging units 784 and 785 used at the point where the pixel data for Rand B in the frame FmIR1 are input and the averaging units 784 and 785used at the point where the pixel data for R and B in the frame FmIR2are input.

The averaging unit 784 averages the pixel data for R within one frame inthe frame FmIR1 to generate average Rave (IR1). The averaging unit 785averages the pixel data for B within one frame in the frame FmIR1 togenerate average Bave (IR1).

The averaging unit 784 averages the pixel data for R within one frame inthe frame FmIR2 to generate average Rave (IR2). The averaging unit 785averages the pixel data for B within one frame in the frame FmIR2 togenerate average Bave (IR2).

The timing for generating the averages Rave (IR1) and Rave (IR2) of thepixel data for R may be shifted from the timing for generating theaverages Bave (IR1) and Bave (IR2) of the pixel data for B so that onlyone averaging unit is required. In such a case, the average generatedfirst is temporarily stored.

The averages Rave (IR1) and Bave (IR1) and the averages Rave (IR2) andBave (IR2) are input into the respective difference calculation units786. The two difference calculation units 786 separately show thedifference calculation unit 786 used at the point where the averagesRave (IR1) and Bave (IR1) are input and the difference calculation unit786 used at the point where the averages Rave (IR2) and Bave (IR2) areinput.

The difference calculation unit 786 calculates the difference betweenthe average Rave (IR1) and the average Bave (IR1) to generate differenceVdf1. The difference calculation unit 786 calculates the differencebetween the average Rave (IR2) and the average Bave (IR2) to generatedifference Vdf2. Although not illustrated in the drawing, thedifferences Vdf1 and Vdf2 are absolutized by an absolute circuit andinput into the difference calculation unit 787.

The difference calculation unit 787 calculates the difference betweenthe difference Vdf1 and the difference Vdf2 to generate differenceVdf12. Although not illustrated in the drawing, the difference Vdf12 isabsolutized by an absolute circuit and input into the comparing unit788.

The comparing unit 788 compares the difference Vdf12 and threshold Th20to generate determination data Ddet2, indicating that visible light isthe predominant light when the difference Vdf12 is equal to or less thanthe threshold Th20, and that infrared light is the predominant lightwhen the difference Vdf12 exceeds the threshold Th20. The determinationdata Ddet2 may also only include two values. The determination dataDdet2 is input into the color gain controller 79.

FIG. 40 shows a state where visible light is the predominant light.Items (a) and (b) of FIG. 40 are the same as items (a) and (b) of FIG.33, respectively. Item (c) of FIG. 40 shows the frames generated basedon the exposures shown in item (b) of FIG. 40. Note that, althoughtiming of the exposures in item (b) of FIG. 40 is shifted from timing ofthe frames in item (c) of FIG. 40 as described in FIG. 8, FIG. 40 showsa state where the respective positions in the time direction (left-rightdirection) coincide with each other while the difference of timing isnot considered.

As is explained in FIG. 33, the exposure amounts AepR, AepG and AepB ofthe imaging unit 3 are substantially the same in the state where visiblelight is the predominant light, and are hardly influenced by thedifference among the sensitivities of R, G and B at the respectivewavelengths IR1 to IR3.

Therefore, as shown in item (d) of FIG. 40, the difference Vdf1generated in the timing of the frame FmIR1 has substantially the samevalue as the difference Vdf2 generated in the timing of the frame FmIR2.Thus, the difference Vdf12 results in a small value.

Item (d) of FIG. 40 shows the respective differences Vdf1 and Vdf2across the respective frames FmIR1 and FmIR2 for convenience ofexplanation. The differences Vdf1 and Vdf2 are actually generated at apredetermined point after the pixel data in the frames FmIR1 and FmIR2are input into the determination unit 78.

FIG. 41 shows a state where infrared light is the predominant light.Items (a) to (c) of FIG. 41 are the same as items (a) to (c) of FIG. 40,respectively. The exposure amounts are influenced by the differenceamong the sensitivities of R, G and B at the respective wavelengths IR1to IR3.

Therefore, as shown in item (d) of FIG. 41, the difference Vdf1generated in the timing of the frame FmIR1 greatly differs from thedifference Vdf2 generated in the timing of the frame FmIR2. Thus, thedifference Vdf12 results in a large value.

The operation of the determination unit 78 is explained in detail belowwith reference to the flowchart shown in FIG. 42. In FIG. 42, thedetermination unit 78 determines whether the frame FmIR1 should begenerated in step S301.

When determining that the frame FmIR1 should be generated now (YES), thedetermination unit 78 averages the pixel data for R in the frame FmIR1so as to generate the average Rave (IR1) in step S302. The determinationunit 78 then averages the pixel data for B in the frame FmIR1 so as togenerate the average Bave (IR1) in step S303.

The determination unit 78 calculates the difference between the averageRave (IR1) and the average Bave (IR1) to generate the difference Vdf1 instep S304.

When determining in step S301 that the frame FmIR1 should not begenerated yet (NO), the determination unit 78 then determines in stepS305 whether the frame FmIR2 should be generated instead.

When determining that the frame FmIR2 should be generated now (YES), thedetermination unit 78 averages the pixel data for R in the frame FmIR2so as to generate the average Rave (IR2) in step S306. The determinationunit 78 then averages the pixel data for B in the frame FmIR2 so as togenerate the average Bave (IR2) in step S307.

The determination unit 78 calculates the difference between the averageRave (IR2) and the average Bave (IR2) to generate the difference Vdf2 instep S308.

When determining in step S308 that the frame FmIR2 should not begenerated yet (NO), the determination unit 78 returns to the processingin step S301.

The determination unit 78 calculates the difference between thedifference Vdf1 and the difference Vdf2 to generate the difference Vdf12in step S309. The determination unit 78 determines whether thedifference Vdf12 is equal to or less than the threshold Th20 in stepS310.

When the difference Vdf12 is equal to or less than the threshold Th20(YES), the determination unit 78 determines that the visible light isthe predominant light instep S311 and returns to the processing in stepS301. When the difference Vdf12 is not equal to or less than thethreshold Th20 (NO), the determination unit 78 determines that theinfrared light is the predominant light in step S312 and returns to theprocessing in step S301. The same procedure is subsequently repeated.

The color gain controller 79 determines whether the visible light is thepredominant light based on the input determination data Ddet2 in stepS201 of FIG. 37.

When the visible light is the predominant light (YES), the color gaincontroller 79 controls the color gain setting unit 62 to select thefirst set of color gains in step S202 and returns to the processing instep S201.

When the visible light is not the predominant light (NO), the color gaincontroller 79 controls the color gain setting unit 62 to select thesecond set of color gains in step S203 and returns to the processing instep S201. The same procedure is subsequently repeated.

<Image Signal Processing Method depending on Determination ofSuperior-Subordinate Relationship of Light Amounts>

The image signal processing method depending on the determination of thesuperior-subordinate relationship of the light amounts executed by theimaging device shown in FIG. 1 is again explained below with referenceto FIG. 43.

In FIG. 43, the imaging unit 3 images the object in the state whereinfrared light with the respective wavelengths IR1, IR2 and IR3 isprojected selectively by the infrared projector 9 so as to generate theimaging signals corresponding to each of R, G and B, in step S501.

The image processing unit 5 generates the data for the respective threeprimary colors R, G and B based on the imaging signals in step S502. Thedetermination unit 78 analyzes the superior-subordinate relationshipbetween the amount of environmental visible light and the amount ofinfrared light projected by the infrared projector 9, in step S503.

The color gain controller 79 controls the color gain setting unit 62 toselect the first set of color gains when the visible light is thepredominant light (YES) in step S504. The image output unit 6 thusoutputs the image signals in step S505 obtained in such a manner as tomultiply the data for the three primary colors by the first set of colorgains.

The color gain controller 79 controls the color gain setting unit 62 toselect the second set of color gains when the visible light is not thepredominant light (NO) in step S504. The image output unit 6 thusoutputs the image signals in step S506 obtained in such a manner as tomultiply the data for the three primary colors by the second set ofcolor gains.

According to the processing from step S504 to step S506, the imagingdevice selects one of the several sets of color gains by which the datafor the three primary colors is multiplied in accordance with theanalyzed superior-subordinate relationship and outputs the image signalsobtained in such a manner as to multiply the data for the three primarycolors by the selected set of color gains.

When the imaging device is set to the intermediate mode, the procedureof processing from step S501 to step S506 is repeated.

<Image Signal Processing Program for Controlling Operation of ImagingDevice Including Determination of Superior-Subordinate Relationship ofLight Amounts>

The image signal processing program when the operation of the imagingdevice including the determination of the superior-subordinaterelationship of the light amounts is controlled by the computer program,is explained below with reference to FIG. 44.

In FIG. 44, the image signal processing program directs the computer instep S601 to obtain the image data based on the imaging signalscorresponding to each of R, G and B generated by imaging the object inthe state where infrared light with the respective wavelengths IR1, IR2and IR3 is projected selectively by the infrared projector 9.

The image signal processing program directs the computer in step S602 toanalyze the relationship between the amount of environmental visiblelight and the amount of infrared light projected by the infraredprojector 9 based on the image data. Although not illustrated in thedrawing, the determination results are stored temporarily in thecomputer.

The image signal processing program directs the computer in step S603 toascertain the stored determination results.

When the visible light is the predominant light (YES), the image signalprocessing program directs the computer in step S604 to control thecolor gain setting unit 62 to select the first set of color gains bywhich the data for the three primary colors generated based on the imagedata is multiplied.

When the visible light is not the predominant light (NO), the imagesignal processing program directs the computer in step S605 to controlthe color gain setting unit 62 to select the second set of color gainsby which the data for the three primary colors generated based on theimage data is multiplied.

According to the processing from step S603 to step S605, the imagesignal processing program controls the color gain setting unit 62 toselect one of the several sets of color gains by which the data for thethree primary colors is multiplied in accordance with the analyzedsuperior-subordinate relationship.

When the imaging device is set to the intermediate mode, the imagesignal processing program directs the computer to repeat the procedureof the processing from step S601 to step S605.

The image signal processing program for controlling the operation of theimaging device including the determination of the superior-subordinaterelationship of the light amounts may also be a computer programrecorded in a computer readable non-transitory storage medium. The imagesignal processing program may also be provided in the state of beingstored in the storage medium or may be provided via a network.

<Example of Switching Sets of Color Gains>

The process of how to switch the sets of color gains used by the colorgain setting unit 62 is explained below with reference to FIG. 45. Items(a) and (b) of FIG. 45 are the same as items (a) and (b) of FIG. 22.

In the normal mode, since visible light is the predominant light, theset of color gains used by the color gain setting unit 62 is constantlythe first set. In the night-vision mode, since infrared light is thepredominant light, the set of color gains used by the color gain settingunit 62 is constantly the second set. The color gain setting unit 62 mayselect the first or second set depending on the mode switched by themode switching unit 72.

In the intermediate mode, the first set is selected when the amount ofvisible light is relatively large, and the second set is selected whenthe amount of infrared light is relatively large, in accordance with thedetermination results of the first or second example of thedetermination method described above.

Therefore, when the mode is switched from the normal mode to theintermediate mode and switched from the intermediate mode to thenight-vision mode, the color gain setting unit 62 uses the first setfrom the normal mode to the middle of the intermediate mode and uses thesecond set from the middle of the intermediate mode to the night-visionmode.

In item (c) of FIG. 45, the intermediate mode is divided into twoperiods for convenience of explanation, wherein the first set isassigned to the period toward the normal mode, and the second set isassigned to the period toward the night-vision mode. The timing ofswitching from the first set to the second set is actually determineddepending on the setting conditions of the thresholds Th10 and Th20 orvarious kinds of other conditions.

The color gain setting unit 62 may hold at least three sets of colorgains so that the third set different from the second set used in theintermediate mode is used in the night-vision mode.

As described above, the intermediate mode and the night-vision mode maybe switched in accordance with the determination results of thesuperior-subordinate relationship of the light amounts. The intermediatemode may be selected when visible light is the predominant light asshown in FIG. 33 and FIG. 40, and the night-vision mode may be selectedwhen infrared light is the predominant light as shown in FIG. 34 andFIG. 41.

Here, the state where visible light is the prominent light and infraredlight is the subordinate light is not necessarily limited to the statewhere the amount of visible light is larger than the amount of infraredlight. Similarly, the state where infrared light is the prominent lightand visible light is the subordinate light is not necessarily limited tothe state where the amount of infrared light is larger than the amountof visible light. The relationship between visible light and infraredlight is not necessarily analyzed on the basis of a particular ratio ofvisible light to infrared light. For example, the relationship betweenvisible light and infrared light may be analyzed in such a manner as toobtain image signals generated with higher resolution.

The present invention is not limited to the present embodiment describedabove, and various modifications and improvements can be made withoutdeparting from the scope of the present invention. For example, theimaging device shown in FIG. 1 may include the infrared projector 9detachably from the casing of the imaging device. The infrared projector9 may be an external component outside of the imaging device. Theimaging device is only required to have a configuration capable ofcontrolling the infrared projector 9 when the infrared projector 9 isattached thereto.

The relationship between visible light and infrared light may beanalyzed based on the measurement results of the light amounts. Thedetermination of the superior-subordinate relationship by thedetermination unit 78 includes the case based on the measurement resultsof the light amounts. The determination of the superior-subordinaterelationship based on the image signals described above tends to easilyreflect the relationship between visible light and infrared light in thevicinity of an object to be imaged. Therefore, the determination of thesuperior-subordinate relationship based on the image signals is betterthan the determination of the superior-subordinate relationship based onthe measurement results of the light amounts.

The controller 7 may change various types of parameters according to therelationship between the amount of environmental visible light and theamount of infrared light. The present embodiment is applicable to thetechnique of changing various types of parameters, without being limitedto the color gains, used for image processing of imaging signals imagedby the imaging unit 3.

The controller 7 may switch imaging modes of various types according tothe relationship between the amount of environmental visible light andthe amount of infrared light. The present embodiment is applicable tothe technique of control of switching various known imaging modeswithout being limited to the intermediate mode or the night-vision mode.The present embodiment is applicable to other imaging techniques usinginfrared light.

The controller 7 and the image processing unit 5 may be composed of aprocessor (CPU) including one or more hardware components. The use ofhardware or software is optional. The imaging device may only includehardware, or part of the imaging device may be composed of software.

As described above, the imaging device according to the embodiment cancapture fine color images even in a state where visible light andinfrared light for night-vision imaging coexist. The image processingdevice, the image signal processing method and the image signalprocessing program according to the embodiment can generate fine colorimage signals based on images captured even in the state where visiblelight and infrared light for night-vision imaging coexist.

What is claimed is:
 1. An image processing device comprising: an imageprocessing unit including a means of generating pixel data for apredetermined color in an intermediate mode implemented based on pixeldata obtained from a predetermined region in a light receiving elementwhen an object is captured in a state where infrared light is projected,and a means of generating the pixel data for the predetermined color ina night-vision mode implemented based on pixel data obtained from awider region than the predetermined region in the light receivingelement when the object is captured in the state where the infraredlight is projected; and a mode stitching unit configured to switch,depending on a condition, between a state where an image output unitoutputs an image signal generated in the intermediate mode and a statewhere the image output unit outputs an image signal generated in thenight-vision mode.
 2. The image processing device according to claim 1,wherein the infrared light is infrared light assigned to thepredetermined color and having a predetermined wavelength.
 3. The imageprocessing device according to claim 1, wherein the wider regionincludes the predetermined region.
 4. The image processing deviceaccording to claim 1, wherein the predetermined region is a regioncorresponding to a color filter of the predetermined color.
 5. The imageprocessing device according to claim 1, wherein the wider regionincludes a region corresponding to a color filter of the predeterminedcolor and a region corresponding to a color filter of a color other thanthe predetermined color.
 6. The image processing device according toclaim 1, wherein the intermediate mode is configured to: generate pixeldata for red based on pixel data obtained from a first region in thelight receiving element when the object is captured in a state where theinfrared light is projected; generate pixel data for green based onpixel data obtained from a second region in the light receiving elementwhen the object is captured in a state where the infrared light isprojected; and generate pixel data for blue based on pixel data obtainedfrom a third region in the light receiving element when the object iscaptured in a state where the infrared light is projected.
 7. The imageprocessing device according to claim 6, wherein: the night-vision modeis configured to generate pixel data for red based on pixel dataobtained from a fourth region in the light receiving element when theobject is captured in a state where the infrared light is projected;generate pixel data for green based on pixel data obtained from a fifthregion in the light receiving element when the object is captured in astate where the infrared light is projected, and generate pixel data forblue based on pixel data obtained from a sixth region in the lightreceiving element when the object is captured in a state where theinfrared light is projected; the fourth region is wider than the firstregion; the fifth region is wider than the second region; and the sixthregion is wider than the third region.
 8. The image processing deviceaccording to claim 1, wherein: the intermediate mode is configured togenerate pixel data for red based on pixel data obtained from a firstregion in the light receiving element when the object is captured in astate where a first infrared light is projected, pixel data obtainedfrom the first region in the light receiving element when the object iscaptured in a state where a second infrared light is projected and pixeldata obtained from the first region in the light receiving element whenthe object is captured in a state where a third infrared light isprojected, generate pixel data for green based on pixel data obtainedfrom a second region in the light receiving element when the object iscaptured in the state where the first infrared light is projected, pixeldata obtained from the second region in the light receiving element whenthe object is captured in the state where the second infrared light isprojected and pixel data obtained from the second region in the lightreceiving element when the object is captured in the state where thethird infrared light is projected, and generate pixel data for bluebased on pixel data obtained from a third region in the light receivingelement when the object is captured in the state where the firstinfrared light is projected, pixel data obtained from the third regionin the light receiving element when the object is captured in the statewhere the second infrared light is projected and pixel data obtainedfrom the third region in the light receiving element when the object iscaptured in the state where the third infrared light is projected; thenight-vision mode is configured to generate pixel data for red based onpixel data obtained from a fourth region in the light receiving elementwhen the object is captured in the state where the first infrared lightis projected, pixel data obtained from the fourth region in the lightreceiving element when the object is captured in the state where thesecond infrared light is projected and pixel data obtained from thefourth region in the light receiving element when the object is capturedin the state where the third infrared light is projected, generate pixeldata for green based on pixel data obtained from a fifth region in thelight receiving element when the object is captured in the state wherethe first infrared light is projected, pixel data obtained from thefifth region in the light receiving element when the object is capturedin the state where the second infrared light is projected and pixel dataobtained from the fifth region in the light receiving element when theobject is captured in the state where the third infrared light isprojected, and generate pixel data for blue based on pixel data obtainedfrom a sixth region in the light receiving element when the object iscaptured in the state where the first infrared light is projected, pixeldata obtained from the sixth region in the light receiving element whenthe object is captured in the state where the second infrared light isprojected and pixel data obtained from the sixth region in the lightreceiving element when the object is captured in the state where thethird infrared light is projected; the fourth region is wider than thefirst region; the fifth region is wider than the second region; and thesixth region is wider than the third region.
 9. The image processingdevice according to claim 1, wherein: the intermediate mode isconfigured to generate pixel data for red based on pixel data obtainedfrom a first region in the light receiving element when the object iscaptured in a state where a first infrared light is projected, generatepixel data for green based on pixel data obtained from a second regionin the light receiving element when the object is captured in a statewhere a second infrared light is projected, and generate pixel data forblue based on pixel data obtained from a third region in the lightreceiving element when the object is captured in a state where a thirdinfrared light is projected; the night-vision mode is configured togenerate pixel data for red based on pixel data obtained from a fourthregion in the light receiving element when the object is captured in astate where the first infrared light is projected; generate pixel datafor green based on pixel data obtained from a fifth region in the lightreceiving element when the object is captured in a state where thesecond infrared light is projected, and generate pixel data for bluebased on pixel data obtained from a sixth region in the light receivingelement when the object is captured in a state where the third infraredlight is projected; the fourth region is wider than the first region;the fifth region is wider than the second region; and the sixth regionis wider than the third region.
 10. The image processing deviceaccording to claim 8, wherein: the first infrared light is infraredlight assigned to the red and having a first wavelength; the secondinfrared light is infrared light assigned to the green and having asecond wavelength; and the third infrared light is infrared lightassigned to the blue and having a third wavelength.
 11. The imageprocessing device according to claim 7, wherein: the fourth regionincludes the first region; the fifth region includes the second region;and the sixth region includes the third region.
 12. The image processingdevice according to claim 6, wherein: the first region is a regioncorresponding to a color filter of the red; the second region is aregion corresponding to a color filter of the green; and the thirdregion is a region corresponding to a color filter of the blue.
 13. Theimage processing device according to claim 7, wherein the fourth region,the fifth region and the sixth region include a region corresponding toa color filter of the red, a region corresponding to a color filter ofthe green and a region corresponding to a color filter of the blue,respectively.
 14. The image processing device according to claim 1,wherein the mode switching unit switches between the state where theimage output unit outputs the image signal generated in the intermediatemode and the state where the image output unit outputs the image signalgenerated in the night-vision mode depending on environmental brightnesswhen the object is captured.
 15. The image processing device accordingto claim 1, wherein the mode switching unit switches between the statewhere the image output unit outputs the image signal generated in theintermediate mode and the state where the image output unit outputs theimage signal generated in the night-vision mode depending on time. 16.An imaging device comprising: the image processing device according toclaim 1; and an imaging unit configured to capture an object.
 17. Animage processing method comprising the steps of: generating pixel datafor a predetermined color in an intermediate mode implemented based onpixel data obtained from a predetermined region in a light receivingelement when an object is captured in a state where infrared light isprojected; generating the pixel data for the predetermined color in anight-vision mode implemented based on pixel data obtained from a widerregion than the predetermined region in the light receiving element whenthe object is captured in the state where the infrared light isprojected; and switching, depending on a condition, between a statewhere an image output unit outputs an image signal generated in theintermediate mode and a state where the image output unit outputs animage signal generated in the night-vision mode.
 18. An image processingprogram recorded in a non-transitory storage medium implementing meansexecutable by a computer, the means comprising: an image processingmeans including a means of generating pixel data for a predeterminedcolor in an intermediate mode implemented based on pixel data obtainedfrom a predetermined region in a light receiving element when an objectis captured in a state where infrared light is projected, and a means ofgenerating the pixel data for the predetermined color in a night-visionmode implemented based on pixel data obtained from a wider region thanthe predetermined region in the light receiving element when the objectis captured in the state where the infrared light is projected; and amode stitching means of switching, depending on a condition, between astate where an image output unit outputs an image signal generated inthe intermediate mode and a state where the image output unit outputs animage signal generated in the night-vision mode.